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Executive Summary 


The scientific and engineering enterprise is itself increasingly global.... International boundaries have 
become considerably less important in structuring the conduct of research and development . 1 

Students in science and engineering (S&E) are preparing for careers in fields where international 
partnerships are increasingly important to advancing knowledge and discoveries. It has been over a 
decade since the National Science Board (NSB) highlighted the importance of international 
collaboration and called for increased government commitment to promoting international S&E 
research and education . 2 NSB called for the National Science Foundation (NSF) to take a leadership 
role in international S&E research and education activities promoting "...increased participation in 
international S&E activities by younger U.S. scientists and engineers from diverse backgrounds, 
especially those in the early stage of their careers, in order to develop an internationally competitive 
and globally-engaged S&E workforce ." 3 

NSF's Office of International Science and Engineering (OISE) helps to further NSF's commitment to 
support the active engagement of early-career S&E researchers in international collaborations. 
Among its programs is the East Asia and Pacific Summer Institutes (EAPSI), which provides 
international fellowships to U.S. graduate students in S&E. 

NSF contracted with Abt Associates to conduct an evaluation of the EAPSI program to investigate 
whether it was meeting its goal of providing U.S. graduate students with international experiences 
that will enable future collaboration with foreign peers. This report presents the findings from this 
evaluation. 

The evaluation found evidence that the EAPSI program is meeting its goals to: 

• introduce U.S. graduate students to East Asia and Pacific science and engineering in the 
context of a research setting; and 

• help students initiate scientific relationships that will better enable future collaboration with 
foreign counterparts. 

The evaluation also found evidence of broader benefits, finding that the program helps to orient 
fellows to the host's society, culture, and language, and provides benefits that extend beyond the 
EASPI fellows. 


National Science Board. 2001. Toward a More Effective Role for the U.S. Government in International 
Science and Engineering. NSB-01-187. National Science Foundation: Arlington, VA., p. 12. 

National Science Board. 2001. 

National Science Board. 2000. Toward a More Effective NSF Role in International Science and Engineering, 
Interim Report. NSB-00-217. National Science Foundation: Arlington, VA., p. 10 
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Key Findings 


To investigate the progress of the program toward its goals, the evaluation paid particular attention 
to the opportunities for international research collaborations, the experiences that develop research 
capacity and global perspectives, and the relationships between U.S. and foreign researchers. 

1. What are the program experiences of program participants and managers? 

2. Does the extent to which former fellows engage in international collaborations differ from 
that of unfunded applicants? 

3. Do fellows' post-award career activities and job characteristics differ from those of 
unfunded applicants? 

4. What are the perceived outcomes of program participation? 

5. Do the outcomes of program participation extend beyond the direct participants? 

The evaluation demonstrated that individuals derive benefits from the program, both on an 
individual and a collective basis. 

Introduce Students to East Asia and Pacific Science and Engineering Research 

Research experiences that included collaborations with foreign scientists at their host sites were 
central to the fellows' EAPSI program experiences. 

• Overall, fellows were satisfied with their EAPSI research activities and interactions with their 
host. All EAPSI fellows would recommend the fellowship to another graduate student seeking an 
international experience, 92 percent of advisors would or have recommended the program to 
other graduate students, and 79 percent of hosts would or have recommended hosting an EAPSI 
fellow to others. 

• EAPSI offers graduate students an opportunity to conduct research in foreign locations and to 
establish relationships that they may build on subsequently in their careers. A majority of 
fellows reported that they were very satisfied with their host's expertise (81 percent) and the 
match between their research interests and the host's (63 percent). 

• A minority of fellows noted difficulties related to their research activities and host interactions. 
Among the difficulties cited were not enough guidance from the host (17 percent), and being 
assigned a role that was less than merited by their skills or knowledge (5 percent). Less than 10 
percent of hosts reported specific challenges regarding collaborating on research with their 
fellow. 

Initiate Relationships That Enable Future International Collaborations 

A key goal of the EAPSI program is to help students initiate scientific relationships that will better 
enable future collaboration with foreign scientists. Although fellows and unfunded applicants 
similarly work with individuals in other countries, the evaluation provides evidence that the EAPSI 
experiences lead to more productive international research collaborations. 
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• Although EAPSI forges new relationships for the fellows, about half of the fellowships build on 
some existing relationships. Only 15 percent of hosts reported that they knew their prospective 
fellow prior to him/her applying. Almost half (45 percent) of the hosts, however, knew the 
graduate advisor of the prospective fellow. 

• Fellows had the opportunity to work with a range of individuals at the host institution. Hosts 
most frequently reported that their fellow worked with the host (80 percent), graduate students 
in the host's research group (66 percent), and other scientists in the host's research group (45 
percent). 

• A majority of the fellows indicated that their relationship with the host has continued beyond 
the fellowship period, through subsequent research collaborations with the host (20 percent), 
and/or additional communications (60 percent). Hosts had similar responses to the fellows, as 
29 percent stated that they had further collaborated with the fellow and 48 percent had 
communications with the fellow post-fellowship. 

• Unfunded applicants continued to pursue other avenues for their international experiences. 
EAPSI fellows and unfunded applicants with PhDs held a similar number of international 
postdoctoral fellowships (0.32 versus 0.28 fellowships, respectively), yet fellows were less likely 
than unfunded applicants to have been employed outside the U.S. (13 versus 39 percent, 
respectively). 

• Forty percent of fellows and 35 percent of unfunded applicants report working with individuals 
located in other countries. Among those who work with a collaborator in a foreign country, 
unfunded applicants were statistically more likely to report that their collaboration included 
joint publications or jointly developed products (82 versus 92 percent). 

• Compared to unfunded applicants, fellows reported a higher number of publications co- 
authored with a foreign collaborator (2.2 versus 1.6 publications) and a larger proportion of 
publications that include a foreign co-author (22 versus 17 percent), suggesting that fellows 
have more productive collaborations with foreign counterparts than do unfunded applicants. 

• Benchmarking analyses comparing career outcomes of EAPSI fellows to national estimates for 
similar science and engineering PhD graduates found that EAPSI fellows were more likely to be 
working with individuals in other countries (80 percent of fellows holding a PhD versus 30 
percent of the SDR sample). 

Extend Benefits Beyond EAPSI Fellows 

The experiences extend beyond the specific participants and seed additional international S&E 

research activities. As such, EAPSI contributes to NSF's efforts promote international S&E among 

early-career scientists and engineers. 

• As reported by host site representatives, host countries become involved in EAPSI to increase 
research collaborations with the U.S., to establish a research network and ongoing relationships 
between scientists, and to provide younger researchers from the U.S. with an opportunity to 
understand the R&D activities in their countries. 
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• About one-third of fellows reported that other researchers from their home institution became 
involved in the research project during the EAPSI fellowships. 

• EAPSI fellows continue to promote international collaborations after they return to the U.S. 
More than half of the fellows engaged in activities where they extended the benefits of their 
EAPSI participation with others, particularly by sharing resources they collected or tools 
developed during their fellowship (59 percent) and teaching others about the research methods 
they learned during that time (56 percent). 

• Fellows were significantly more likely than unfunded applicants to engage in a series of 
activities to foster international collaborations among others (40 percent versus 30 percent, 
respectively). 

• Host site representatives identified several benefits of the program for those involved— fellows, 
other students in the host lab or institution, host researchers, and the host institutions and 
countries. Respondents commonly cited increased opportunities for collaboration and 
networking as benefits of participation. 

• Host researchers most commonly reported that they agreed to serve as a host in the program 
because of a shared interest in the prospective fellow's research project (64 percent), an 
interest in creating an international environment in their research group (59 percent), and to 
establish or maintain collaboration with a U.S. researcher (57 percent). 

• As a result of the EAPSI experience, most hosts identified some benefit; only 11 percent 
perceived no personal benefits from their participation. The most common benefits selected by 
respondents included enhanced interest in collaborating with U.S. researchers (31 percent), 
established or renewed collaborations with other U.S. researchers (30 percent), and published 
papers based on the research conducted (25 percent). 


Additional Findings 


The evaluation was also designed to address the following questions: 

1. What are the characteristics of people who apply for and participate in the EAPSI program? 

2. What motivates individuals to apply for and participate in the program, and what are 
individuals' experiences during the application process? 

The EAPSI applicants represent graduate students with varied backgrounds and motivations for 
participating in the program. 

• Applicants most commonly applied to EAPSI to enhance their skills or knowledge as a researcher 
(82 percent). 

• The majority of EAPSI applicants did not have extensive academic and travel experiences outside 
the U.S. at the time of application. Thus, through the research opportunities, EAPSI introduced 
fellows to international experiences. Only 36 percent of applicants participated in study abroad 
programs as undergraduates or graduate students, and 18 percent had attended elementary or 
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secondary school outside the U.S. Around one-third had lived outside the U.S. for six months or 
longer (35 percent) or visited or lived in one of the seven EAPSI host locations for a month or 
longer (30 percent). Less than a quarter of applicants (22 percent) reported that they had 
attended a research conference outside the U.S. 

• A majority of hosts agreed that that their research interests were well-matched with those of 
their fellow (82 percent), and that they and the fellow shared similar goals and expectations for 
the EAPSI experience (60 percent). 

Applicants are also drawn to EAPSI in part because it provides an opportunity to learn about the 
society, culture, and language of host sites. Nearly all fellows reported participating in cultural and 
leisure activities while abroad, and a larger majority reported becoming comfortable with the 
traditions and culture of their host site. 

• Applicants commonly reported applying to EAPSI in part to learn about the culture, history, and 
geography of another country (77 percent), and to conduct research with a specific person or at 
a specific institution (70 percent). 

• Nearly all fellows participated in cultural and leisure activities in their host site which included 
sightseeing (97 percent), exploring the landscape or geography (91 percent), and visiting 
museums (89 percent). 

• Among fellows placed in primarily non-English speaking host sites— Japan, Korea, Taiwan, and 
China— 67 percent reported some type of language training in preparation for their 
international fellowship experience. This included self-guided study (47 percent), a formal 
language training course (23 percent), and working with a conversational partner or tutor (9 
percent). 

• Communication or language issues were the most frequently cited difficulties that fellows 
encountered during their fellowship; however, this varied significantly by host site where a 
greater number of fellows in primarily non-English speaking host sites experienced these 
difficulties (28 to 44 percent within these sites) than fellows in primarily English-speaking host 
sites— New Zealand, Australia, and Singapore (0 to 10 percent within these sites). 


Evaluation Approach 


The evaluation was designed to assess how the EAPSI program contributes to the engagement of 
early-career S&E researchers in international research collaborations. The study incorporated both 
extant and primary data sources. These included surveys administered to program applicants (both 
those who received EAPSI fellowships and those who did not), EAPSI foreign hosts, and EAPSI 
graduate advisors, as well as telephone interviews with EAPSI agency officials and managers at 
foreign counterpart agencies. The study included both descriptive and comparative analyses, 
including a rigorous quasi-experimental design to answer the questions about program impacts. 

Propensity score analysis (PSA) was used to construct groups of awardees and non-awardees that 
were statistically similar across a number of pre-existing characteristics, in order to compare the 
outcomes of fellows to those of unfunded applicants, using pre-award characteristics of applicants 
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to mitigate the potential threat of selection bias. To situate the EAPSI program participants' and 
applicants' outcomes within the national S&E context, a secondary set of comparative analyses was 
conducted using data from the Survey of Doctoral Recipients (SDR) and the National Survey of 
Recent College Graduates (NSRCG). The evaluation also used descriptive analyses to explore the pre- 
award international research experiences and other characteristics of EAPSI applicants and host 
scientists; to understand what motivated STEM graduate students to apply for an EAPSI fellowship 
and what led EAPSI host scientists to host a fellow, the perceptions and role of EAPSI advisors, and 
to describe the experiences of EAPSI participants (i.e., fellows and hosts), both during and after the 
period of the fellowship. 


Abt Associates Inc. 


Executive Summary 6 


Evaluation of the East Asia and Pacific Summer Institutes Program: Final Report 


1 Introduction 


The scientific and engineering enterprise is itself increasingly global. ... International boundaries have 
become considerably less important in structuring the conduct of research and development . 4 

Students in science and engineering (S&E) are preparing for careers in fields where international 
partnerships are increasingly important to advancing knowledge and discoveries. It has been over a 
decade since the National Science Board (NSB) highlighted the importance of international 
collaboration and called for increased government commitment to promoting international S&E 
research and education. 5 NSB called for the National Science Foundation (NSF) to take a leadership 
role in international S&E research and education activities promoting "...increased participation in 
international S&E activities by younger U.S. scientists and engineers from diverse backgrounds, 
especially those in the early stage of their careers, in order to develop an internationally competitive 
and globally-engaged S&E workforce." 6 

The Office of International Science and Engineering (OISE) serves as a primary hub within NSF for 
international research opportunities for United States (U.S.) scientists and engineers. OISE's East 
Asia and Pacific Summer Institutes (EAPSI) program, which provides international fellowships to U.S. 
graduate students in S&E, is one program which helps to further NSF's commitment to support the 
active engagement of early-career S&E researchers in international collaborations. Highlights 
generated by NSF that describe specific EAPSI fellowships are included in Appendix A. 

In August 2009, NSF contracted with Abt Associates to conduct an evaluation of the EAPSI program 
to investigate whether it was meeting its goal of providing U.S. graduate students with international 
experiences that will enable future collaboration with foreign peers. The study involved applicants 
for the 2000 to 2009 summers, , their advisors and foreign hosts, and the program officials who are 
familiar with EAPSI in each of the seven program locations. This report presents the findings from 
the evaluation. 

Chapter 1 of this report describes the EAPSI program and situates it within recent trends in 
international scientific research. The report then describes the methodology used to conduct the 
evaluation (Chapter 2); characteristics of EAPSI applicants as well as their and the hosts' motivations 
for participation (Chapter 3); respondents' experiences with and perceptions of the program 
(Chapter 4); outcomes of the program (Chapter 5); and conclusions and implications of the 
evaluation findings. The appendices included with this report provide additional details: Appendix A 
contains NSF-generated descriptions of specific EAPSI fellowships, Appendices B through D describe 
methodological details, Appendix E contains the survey instruments, and Appendix F contains a 
comparison of EAPSI applicants to national data on career outcomes. 


National Science Board. 2001. Toward a More Effective Role for the U.S. Government in International 
Science and Engineering. NSB-01-187. National Science Foundation: Arlington, VA., p. 12. 

National Science Board. 2001. 

National Science Board. 2000. Toward a More Effective NSF Role in International Science and Engineering, 
Interim Report. NSB-00-217. National Science Foundation: Arlington, VA., p. 10 


Abt Associates Inc. 


1. Introduction 7 




Evaluation of the East Asia and Pacific Summer Institutes Program: Final Report 


1.1 The EAPSI Program 


EAPSI provides a $5,000 stipend and travel costs to and from the host site to support U.S. graduate 
students to spend the summer (eight to ten weeks) conducting research in Australia, China, Japan, 
Korea, New Zealand, Singapore, or Taiwan 7 ; hosting organizations provide additional support to 
cover living expenses (training, food, and lodging) in the host country. The program is designed to 
immerse U.S. scholars in the scientific and social culture of the host location. Fellows are 
encouraged to visit other institutions within the host country, to get a broader perspective on its 
science and engineering enterprise. Since the program's inception, over 2,000 U.S. graduate 
students have participated in the program. 

The program logic model (Exhibit 1.1) identifies the links between program processes and outcomes, 
and helped ground the evaluation in the program's theory. As depicted in the logic model, the goals 
of the EAPSI program are to "to introduce U.S. graduate students to East Asian science and 
engineering in the context of a research setting, and to help students initiate scientific relationships 
that will better enable future collaboration with foreign counterparts." The programmatic strategy 
for meeting these goals is to provide U.S. graduate students with first-hand experience conducting 
research in one of the participating countries of East Asia and the Pacific; an introduction to science, 
science policy, and scientific infrastructure of the host location; and a chance to experience the 
society, culture, and language of the host country. 8 

Funding for the EAPSI program has grown substantially over the past decade, from approximately 
$500,000 in 2000 to almost $2 million in 2009. The award amount and the number of awards have 
also increased over the same time period, from $2500 to $5000 and from 140 to 195, respectively. 
The program has expanded in scope over the years; for instance, when the summer experiences 
program launched in 1990, Japan was the only host country. Since then the number of host 
countries has gradually increased with the addition of Korea in 1995 Taiwan in 2000, China and 
Australia in 2004, New Zealand in 2007, and Singapore in 2009. The fellowship period has remained 
the same since the beginning of the program: each institute, except for Japan, lasts for 
approximately eight weeks; the Japan institute lasts approximately 10 weeks during the summer. 


7 The program was not referred to as EAPSI until Australia became a host site in 2004. Prior to 2004, the 
name of the program changed each time a new host country joined the program. For example, the 
program was called Summer Program in Japan from 1990 to 1994 until Korea joined in 1995 and the name 
changed to Summer Programs in Japan and Korea. 

8 National Science Foundation. 2010. East Asia and Pacific Summer Institutes for U.S. Graduate Students 
(EAPSI) program solicitation, http://www.nsf.gov/pubs/2010/nsfl0591/nsfl0591.htm 
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Exhibit 1.1: EAPSI Logic Model 



EAPSI PROGRAM GOALS 

• To introduce early-career scientists and engineers to international 
collaborative research opportunities 

• To further their research capacity and global perspective 

• To forge long-term relationships between U.S. and foreign scientists, 
engineers, and technologists 


EAPSI PROGRAM ACTIVITIES 

NSF 

• Development of program solicitation 

• Selection of awardees 

• Program monitoring 

Fellows 

• Development of application and selection of international host institution 

• Conducting research projects at host institution 

• Submission of annual and final reports 

Foreign host institutions 

• Provision of facilities, research environment, and mentoring 


EAPSI PROGRAM OUTPUTS 

• Final reports 

• Joint publications, patents, funding applications 

• Tools, data sets, samples, instruments 

• Research projects suitable for continued partnership 


EAPSI PROGRAM OUTCOMES 

• Participant familiarity with international science and engineering enterprise 

• Familiarity with host country's research enterprise, language, and customs 

• Career choices involving opportunities for international collaboration 

• Advancement of U.S. and foreign scientists' research agendas 

• Development of ties between U.S. institutions and foreign hosts 

• Awareness of opportunities for international collaboration in participant 
institutions 


EAPSI PROGRAM IMPACTS 

• Emergence of a cadre of scientists and engineers who will play a leadership 
role in forging international collaborations 

• Development of collaborations that are of great value to the nation 

• Maintenance of U.S. leadership in science and technology 

• Research capacity building in foreign institutions 
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1.2 Globalization of Science and Engineering 


A decade ago, the NSB noted that "international boundaries have become considerably less 
important in structuring the conduct of research and development" in S&E fields. 17 Ten years later, 
the globalization of S&E research and education continues. 9 Recent data show that while the U.S. is 
still a major global force in S&E research and education, other countries are increasing their 
competiveness in these areas. For example, countries are increasingly competing with the U.S. for 
foreign students, and top student destinations include countries such as the United Kingdom, 
Germany, and France; the proportion of foreign students in the U.S. decreased from 25 percent in 
2000 to 20 percent in 2006. 10 Additionally, other countries are producing more S&E graduates than 
the United States; of the more than 4 million first university S&E degrees awarded worldwide in 
2006, 21 percent were earned by students in China, 19 percent by students from the European 
Union, and only 11 percent by students in the U.S. * 11 

Data also show that foreign nations are placing a great effort on developing a skilled S&E workforce 
that can compete internationally. For instance, between 1997 and 2007, the estimated number of 
S&E researchers in the U.S. grew by 40 percent to reach approximately 1.4 million. Flowever, over 
the same time period, the number of S&E researchers in China grew by 173 percent to also reach 1.4 
million. 12 Additionally, while the U.S. remains the leader in S&E research and development 
expenditures (accounting for 33 percent of total spending), countries such as China are investing 
large amounts of funding in their own R&D expenditures (averaging 19 percent annually over the 
past decade), and the U.S.' R&D/GDP ratio now ranks only eighth among economies tracked by the 
OECD. 13 

Reflecting the importance of these trends and the associated opportunities and challenges for the 
U.S., a companion publication, which highlighted the globalization of science and engineering 
research, was prepared for the Science and Engineering Indicators 2010. u Further, the Committee 
on Global Science Policy and Science Diplomacy of the National Research Council convened a 
workshop to discuss the importance of international science engagement and global science 


National Science Board. 2012. Science and engineering indicators 2012. NBS-12-01. National Science 
Foundation: Arlington, VA. 

10 National Science Board. 2010a. Science and engineering indicators 2010. NBS-10-01. National Science 
Foundation: Arlington, VA. 

11 National Science Board. 2010a. 

12 National Science Board. 2010a. 

13 National Science Board. 2010a. 

14 National Science Board. 2010b. Globalization of science and engineering research: A companion to 
science and engineering indicators 2010. NBS-10-3. National Science Foundation: Arlington, VA. 
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cooperation, and the potential of science policy and science diplomacy in meeting international 
challenges. 15 

In light of these trends, it is important that the U.S. fully engage in international collaborations. By 
establishing international networks of scientists, resources can be shared and ideas can be 
developed, tested, and implemented across traditional boundaries. 16 These partnerships can also 
serve as a powerful tool of international diplomacy. As the NSB explains, "science and engineering 
partnerships can strengthen international relationships and. ..promote basic scientific values such as 
accountability, meritocracy, transparency, and objectivity," and policymakers have noted that the 
inclusion of developing nations in these collaborative efforts can promote self-sufficiency and 
encourage international participation in a variety of areas. 17 


1.3 Programs Promoting International Research Collaboration 


Funding for international science and engineering partnerships is concentrated in three federal 
agencies: the Department of Defense (DOD), the National Institutes of Health (NIH), and NSF (NSB 
2008). 18 Within NSF, OISE serves as a focal point for international science and engineering activities 
both inside and outside NSF. OISE supports programs to expand and enhance leading-edge 
international research and education opportunities for U.S. scientists, engineers, and graduate 
students especially at the early-career stage. It works to build and strengthen effective institutional 
partnerships throughout the global science and engineering research and education community, 
and it supports international collaborations in NSF's priority research areas. OISE is housed within 
the Office of the NSF Director since its role is Foundation-wide. The Office carries out its functions 
through close partnership with the NSF Directorates and through its own program activities. 

OISE administers the East Asia and Pacific Summer Institutes (EAPSI) program in order to offer early- 
career researchers an opportunity to forge collaborative relationships with foreign scientists and 
engineers. Other NSF programs that facilitate short-term international collaborations include the 
Pan-American Advanced Studies Institutes (PASI) jointly sponsored with the Department of Energy, 
which funds students or junior researchers to participate in short courses designed to disseminate 
knowledge and stimulate training and cooperation among researchers from countries in the 
Americas; 19 and the Catalyzing New International Collaborations program, which provides funding 


National Research Council, Committee on Global Science Policy and Science Diplomacy. 2011. U.S. and 
International Perspectives on Global Science Policy and Science Diplomacy: Report of a Workshop. National 
Academies Press, Washington, DC. 

16 National Science Board. 2008. International Science and Engineering Partnerships: A Priority for U.S. 
Foreign Policy and Our Nation's Innovation Enterprise. NSB-08-4. National Science Foundation: Arlington, 
VA. 

17 National Science Board. 2008 

18 National Science Board. 2008. 

19 National Science Foundation. 2006a. Pan-American Advanced Studies Institutes Program (PASI) (NSF 10- 
517). Retrieved from http://www.nsf.gov/pubs/2010/nsfl0517/nsfl0517.pdf 


Abt Associates Inc. 


1. Introduction 11 




Evaluation of the East Asia and Pacific Summer Institutes Program: Final Report 


for short-term activities such as planning visits, workshops, initial data-gathering activities, and the 
development of research network. 20 

Another program sponsored by NSF that aims to promote long-term international collaborative 
research is the Partnerships for International Research and Education (PIRE) program. This program 
funds proposals from all areas of NSF-supported science in an effort to: enhance research excellence 
through international partnerships; promote educational excellence via international collaborations; 
and strengthen U.S. capacity for international engagement. 21 NSF also sponsors the International 
Research Fellowship Program (IRFP), which provides support to postgraduate scientists (generally a 
year or two after the receipt of a doctoral degree) for a 9- to 24-month research experience 
abroad. 22 

Other U.S. agencies and organizations recognize the value of international collaborations, and have 
therefore implemented programs to facilitate the international exchange of S&E research. For 
instance, the National Institutes of Health (N I H ) funds the Fogarty International Center, whose 
programs support global health research conducted through international partnerships, build 
partnerships between research institutions in the U.S. and abroad, and train the next generation of 
scientists to address global health issues. 23 

Available evaluation information for these programs, although limited, suggests that participating 
students reported gaining new knowledge and skills during their international collaboration. Many 
also reported that they would be likely to engage in international collaborations in the future, 
perhaps as a result of their international research experience. For instance, an evaluation of NSF's 
International Research Experience Program (which ended in 2008) found that students who engaged 
in an international research experience reported gaining technical, communication, and language 
skills; developed an appreciation for cultural differences; and felt that their research experience 
would "create opportunities for future international collaboration." 24 An evaluation of the Research 
Internships in Science and Engineering (RISE) program, which provides funding for undergraduate 
students to complete summer internships in German higher education institutions, reached similar 
conclusions. Researchers found that the majority of RISE student participants developed a greater 
understanding of German culture, the vast majority reported an increased desire to travel abroad, 


20 National Science Foundation. 2006b. Catalyzing New International Collaborations (NSF 11-508). Retrieved 
from http://www.nsf.gov/pubs/2011/nsfll508/nsfll508.pdf 

21 National Science Foundation. 2006c. Partnerships for International Research and Education (PIRE) (NSF 
09-505). Retrieved from http://www.nsf.gov/pubs/2009/nsf09505/nsf09505.pdf 

22 National Science Foundation IRFP Program Solicitation. 
http://www.nsf.gOv/pubs/2006/nsf06582/nsf06582.html#pgm_desc_txt 

23 The John E. Fogarty International Center: U.S. Department of Hea Ith and Fluman Services, National 
Institutes of Hea Ith . (n.d.). Fogarty at 40: Advancing Science for Global Health. Retrieved from 
http://www.fic.nih.gov/news/publications/fogarty_40th_brochure.pdf 

24 Spencer, D. 2008. International research experience program: International research opportunities for 
students at NSF science and technology centers. Retrieved August 18, 2011 at: 
http://66.116.177.96/IREP%20Evaluation%20Report.pdf 
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and about 1 in 4 alumni reported returning to Germany at some point in the future, either to visit or 
to pursue work/study opportunities. 25 Finally, an evaluation of NSF's International Research and 
Education in Engineering (IREE) pilot program also found that program participants, especially 
graduate and postgraduate students, reported acquiring new research skills, and that they planned 
to continue collaborations with their international counterparts at the end of their program 
experience. 26 


1.4 Purpose of this Study 


The importance of international collaboration cannot be overstated. Promoting international 
engagement at all levels is crucial to fostering successful research partnerships and developing the 
next generation of S&E researchers. As preliminary evaluation data demonstrate, providing 
graduate students and junior researchers with an opportunity to engage in an international research 
experience may help them improve their own research capabilities and encourage them to pursue 
future collaborations with their peers abroad. This, in turn, could lead U.S. researchers to reap 
benefits such as increased visibility in the research community, access to more substantial funding 
and resources, and the opportunity to benefit from the expertise of international colleagues. 

In the climate of budget cuts, federal agencies are under more pressure to demonstrate the value of 
their programs. To assess the contribution of the NSF EAPSI program to the engagement of early- 
career S&E researchers in international research collaborations, the evaluation was designed to 
answer the following questions: 

1. What are the characteristics of people who apply for and participate in the EAPSI program? 

2. What motivates individuals to apply for and participate in the program, and what are 
individuals' experiences during the application process? 

3. What are the program experiences of program participants and managers? 

4. What are the perceived outcomes of program participation? 

5. Do fellows' post-award career activities and job characteristics differ from those of 
unfunded applicants? 

6. Does the extent to which former fellows engage in international collaborations differ from 
that of unfunded applicants? 

7. Do the outcomes of program participation extend beyond the direct participants? 


The Institute of International Education. 2009. Evaluating the DAAD's Research Internships in Science and 
Engineering (RISE) Program: A Final Report. 

Flattau, P.E., Lai, B., Laskey, A., and Ford, J. J. 2009. Portfolio Evaluation of the National Science 
Foundation's Grants Program on "International Research and Education in Engineering" (IREE). Institute 
for Defense Analyses, Science & Technology Policy Institute: Washington, DC. 
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2 Methodology 


It was important for this study to both measure individuals' experiences with the program and to 
investigate the effects of the program on participants, particularly with respect to their international 
research collaborations. As detailed below, the study used a mixed methods design, incorporating 
extant and primary data sources, to answer the specific research questions. Further, the evaluation 
included both descriptive and comparative analyses. The descriptive analysis provided summary 
information about the characteristics, experiences, and perceptions of individuals involved with the 
program, as well descriptive comparisons of differences between groups or points in time. The main 
comparative analyses were designed to investigate the effects of the program on participants, and 
specifically attribute any observed differences to participation in EAPSI. Thus, rigorous quasi- 
experimental impact analyses compared the outcomes of fellows to those of unfunded applicants, 
using pre-award characteristics of applicants to mitigate the potential threat of selection bias. A 
secondary set of comparative analyses between EAPSI applicants (and fellows) to a nationally 
representative sample of STEM doctorates from the Survey of Doctoral Recipients (SDR) was used to 
situate the outcomes of EAPSI program participants and applicants within the national S&E context. 

This study was designed to evaluate the extent to which NSF's EAPSI program introduces U.S. 
graduate students to foreign research setting and helps initiate scientific relationships that enable 
future collaboration with foreign scientists. 


2.1 Data Sources 


Data for the evaluation were drawn from extant program records. In addition, surveys were 
administered to program applicants (both those who received EAPSI fellowships and those who did 
not), EAPSI foreign hosts, and EAPSI graduate advisors. Telephone interviews with EAPSI managers 
at foreign counterpart agencies were also conducted. Details for each source are provided below. 

2.1.1 Extant Data 

Extant data came from NSF's administrative records on applicants from the Survey of Doctoral 
Recipients (SDR) and the National Survey of Recent College Graduates (NSRCG). NSF's administrative 
records on applicants were used to construct the study sample. Existing national data from the SDR 
and from NSRCG were used to compare the characteristics and outcomes of the EAPSI applicants 
and awardees to the national samples with the similar degrees in the STEM fields. 

2.1.2 Primary Data 

The core data for the evaluation were gathered through online surveys— completed from January to 
March, 2011— of EAPSI applicants, hosts and graduate advisors. The surveys gathered information 
about experiences prior to, during, and after the EAPSI program. The EAPSI applicant survey had 
some modules that were specific to either unfunded applicants or EAPSI fellows. Copies of this 
survey, the EAPSI foreign host survey and EAPSI graduate advisors survey are included in Appendix 
E. 

Telephone interviews were also conducted in February and March 2011 with EAPSI host site 
representatives— individuals who were involved in the management and organization of EAPSI in 
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each of the host locations. These individuals play an important role in the implementation of EAPSI 
and their perspectives on the program were important to document. Interview respondents 
included two groups. The first was agency officials who worked in the foreign agencies that oversee 
the EAPSI program (the Australian Academy of Science, Chinese Ministry of Science and Technology, 
Japan Society for the Promotion of Science, Korea Science and Engineering Foundation, Royal 
Society of New Zealand, National Research Foundation of Singapore, and National Science Council of 
Taiwan). The second group included program managers responsible for running the EAPSI program 
in each of the locations (the EAPSI program in Australia, Summer Institute in China, the Summer 
Program in Japan, Korea Summer Institute Program, EAPSI program in New Zealand, EAPSI summer 
program in Singapore, and Summer Institute in Taiwan). In two locations, a single individual served 
as both the agency official and the program manager. 


2.2 Study Sample 


The target populations for the study included all individuals who had applied to the EAPSI program 
to conduct fellowships during the 2000 through 2009 summers, fellows' graduate advisors, and the 
research scientists who served as foreign hosts during this period. These samples are described 
below. 

EAPSI fellows and unfunded applicants. All individuals who applied to the EAPSI program to 
conduct fellowships during the 2000 through 2009 summers were included in the EAPSI study. 27 
Some individuals applied to the program multiple times, but for the purposes of the study they were 
given a single status as follows: the applicants who had ever received a fellowship were considered 
awarded and the applicants who never received a fellowship were considered unfunded. If 
individuals received more than one award, they were included in the study for their most recent 
award. 

Exhibit 2.1 shows the sample size and response rate for the applicant survey, which included 
individuals who applied to conduct fellowships during the 2000 through 2009 summers. The 
universe of study-eligible applicants included 1,298 and 811 unfunded applicants (total n=2,109 
applicants). Thirty-five applicants (15 awardees and 20 non-awardees) were not eligible for the 
study and were excluded from the sampling frame. 28 Eliminating these applicants reduced the 
eligible sample to 2,074 applicants (1,283 awardees and 791 non-awardees). The overall response 
rate for the applicant survey was 63 percent. The response rate for the awardees was 73 percent 
(n=938) and for unfunded applicants 46 percent (n=365). 


A previous study of the program, conducted by Westat in 2002, examined the fellowship experiences and 
outcomes of participants in the Japan programs between 1988 and 1999. Butler, P. Glover, D., Miyaoka, 

A., Bennett-Harper, S., & Sproul, F. 2002. Evaluation of the National Science Foundation's Japan Programs. 
Westat. 

Individuals were excluded for one of the following reasons: individual was an advisor not an applicant; 
individual was awarded the EAPSI fellowship but declined the award; individual did not recall applying for 
an EAPSI fellowship; or inspection of data showed that the individual was not eligible to have applied for 
EAPSI or to participate in the study. Although it is likely that these data were erroneous, it was not 
possible to verify or correct the information provided. 
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Exhibit 2.1: Final Sample Size and Response Rates for the EAPSI Applicant Survey 


Overall 

Awarded 

Unfunded 

a. Target Sample 3 

2,109 

1,298 

811 

b. Final Survey Sample 6 

2,074 

1,283 

791 

c. Number of Completed & Partially 0 Completed Surveys 

1,326 

948 

378 

d. Number of Completed Surveys 

1,303 

938 

365 

Response Rates 

e. Response Rate (d/b) 

63% 

73% 

46% 


a The target sample was those individuals who applied to conduct fellowships in the 2000 through 2009 summers. 
b 35 individuals were determined to be ineligible for the EAPSI study. 

c 23 respondents were classified as 'partial' responders because they had completed less than 15 percent of the 
questions on the survey. 

EAPSI hosts. All hosts of eligible awardees who were identified in NSF records were included in the 
evaluation, resulting in 1,125 individuals. Eighteen hosts were not eligible for the study and were 
excluded from the sampling frame. 29 Eliminating these hosts reduced the eligible sample to 1,107 
hosts. The overall response rate for the host survey was 61 percent (n=671). 


Exhibit 2.2: Final Sample Size and Response Rates for the EAPSI Host Survey 


I Overall 

a. Target Sample 

1,125 

b. Final Survey Sample 3 

1,107 

c. Number of Completed & Partially Completed Surveys 6 

677 

d. Number of Completed Surveys 

671 

Response Rates 

e. Response Rate (d/b) 

61% 


a 18 individuals were determined ineligible for the EAPSI study. 

b 6 respondents were classified as 'partial' responders because they had completed less than 20 percent of the 
questions on the survey. 


Advisors of EAPSI fellows 30 . U.S. advisors of EAPSI fellows who provided a reference for the EAPSI 
application in years starting in 2004 to 2009, and for whom records were available, were included in 
the study. The universe included 925 study-eligible advisors. Twenty-one advisors were not eligible 
for the study and were excluded from the sampling frame. 31 Eliminating these advisors reduced the 
eligible sample to 904 advisors. The overall response rate for the advisor survey was 71 percent 
(n=644). 


29 Individuals were excluded for one of the following reasons: individual was deceased; individual was 
hosting an applicant that was awarded the EAPSI fellowship but had declined the award; individual 
reported not hosting an EAPSI fellow; or inspection of data showed that the individual was not eligible to 
participate in the study. Although it is likely that these data were erroneous, it was not possible to verify 
or correct the information provided. 

30 EAPSI applicants were not required to provide the names of advisors prior to 2004. 

31 Individuals were excluded for one of the following reasons: individual was not the awardees' advisor; 
individual was deceased; individual was the advisor to an applicant that was awarded the EAPSI fellowship 
but had declined the award; or inspection of data showed that the individual was not eligible to 
participate in the study. Although it is likely that these data were erroneous, it was not possible to verify 
or correct the information provided. 
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Exhibit 2.3: Final Sample Size and Response Rates for the EAPSI Advisor Survey 


Overall 

a. Target Sample 

925 

b. Final Survey Sample 3 

904 

c. Number of Completed Surveys' 3 

644 


Response Rates 

d. Response Rate (c/b) 71% 

a 21 individuals were determined ineligible for the EAPSI study. 
b There were no partially (less than 20 percent) surveys from this respondent group. 

EAPSI host site representatives (agency officials and program managers). For each EAPSI host 
location, NSF identified one or two individuals for interviews —typically one agency official and one 
program manager— who were most familiar with the EAPSI program. In some host locations a single 
individual served in both roles. Interviews with ten host site representatives were conducted 
between February and March, 2011. Eleven potential respondents were invited, and a total of ten 
representatives, at least one individual from each host location, were interviewed by the research 
team. 


2.3 Non-Response 


An initial step in the analysis was to explore the consequences of survey non-response, which could 
lead to bias if the awarded or unfunded applicants who did not participate in the study would have 
given systematically different responses to the survey than the individuals who participated in the 
survey. Two types of non-response were investigated: unit non-response, where no survey was 
completed; and item non-response, where individual items of an otherwise completed survey were 
missing. 

2.3.1 Unit Non-Response 

To address unit non-response for the applicant survey, information from NSF program records was 
used to estimate the probability that a person would respond to the survey, as a function of baseline 
characteristics that were available (e.g. proposal score, cohort year, gender). These probabilities 
were used to create weights that were then used to adjust estimates to alleviate the potential bias 32 
due to non-response. This method is described in more detail in Appendix B. It was not possible to 
conduct a non-response bias analysis for the host and advisor samples, as no appropriate extant 
data were available. Hence findings described in this report refer only to the survey respondents, 
and not to all EAPSI hosts or advisors. 

2.3.2 Item Non-Response 

Item non-response refers to information missing on one or more specific questions in an otherwise 
completed survey. Since the amount of missing data on an individual item was modest (less than 5 
percent) across all returned surveys, it was assumed that data on an item were missing at random. 


Note that a large non-response rate does not necessarily create bias. For example, if the non-respondents 
were similar across the awardees and non-awardees, then the impact estimate would not be biased 
necessarily; rather, any effect of the program could not be generalized to the non-respondents (i.e. it 
would create an external validity problem but not necessarily an internal validity issue). 
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Statistics on missing data are presented in all exhibits in this report. Where appropriate, the study 
imputed missing covariate values; outcome variables were not imputed (for more details on item 
non-response analysis see Appendix B). 


2.4 Analyses 


A series of analyses were conducted to answer the descriptive, comparative, and impact study 
questions, as described below. 

2.4.1 Descriptive Analyses 

Most of the research questions were addressed through the use of simple descriptive statistics such 
as means and percentages, as well as cross-tabulations to illustrate patterns of responses for 
groups, or the distribution across subgroups of interest. In general, categorical variables were 
summarized in terms of the percentage of respondents who indicated a particular response, and 
continuous outcomes were presented in terms of means. All exhibits include information on the 
total number of respondents on which percentages or means were computed as well as information 
on item non-response. Estimates were adjusted using weights to account for unit non-response (to 
mitigate any potential bias), so that parameter estimates are representative of the EAPSI program as 
a whole. 

The qualitative data from open-ended survey questions, for example survey items that asked 
respondents to describe their individual experiences and perceptions, were examined for common 
themes and standard coding techniques were applied, where appropriate. These responses provide 
detailed examples of individual experiences, but are not generalizable, because it is not possible to 
tell to what extent the view described is shared by others . 33 Specifically, those who chose to write 
an open-ended response may have systematically different views than those who chose to leave 
such items blank (for example, be extremely satisfied or unsatisfied with the experience). Responses 
to survey items that included an "other, please specify" option were coded by the study team for 
classification into one of the existing response options, whenever possible. 

The qualitative data from the interviews were examined across respondents. Coding started with a 
priori themes, but was iterative as additional themes, which emerged during the initial coding, were 
then applied across all cases as appropriate. The analysis of the interview data explored meaningful 
patterns of similarities and differences, such as the level and nature of host country involvement, 
satisfaction with the fellows and the program, and program-related challenges. 

2.4.2 Impact Analyses 

The evaluation was designed to answer the following research questions about the impact of the 
EAPSI award on participants: 

• Do fellows' international collaborations and other international activities differ from those of 
unfunded applicants in frequency, length, or type of activities? 


The spelling of words in open-ended comments provided by respondents and presented as quotes in this 
report were corrected where necessary. 
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• What are the fellows' post-award career activities and job characteristics? How do these 
compare to unfunded applicants and the national samples? 

The goal of the impact component of the evaluation was to estimate the effect of EAPSI on its 
participants. If EAPSI brings about changes in its participants, then these individuals should have 
different outcomes, post-participation, than they would have had in the absence of program 
participation. Although a random assignment study would have allowed a more rigorous test of the 
causal impact of the EAPSI award on its recipients, this design was not feasible since awardees had 
already been selected based on the merits of their EAPSI proposal. Instead, a rigorous quasi- 
experimental design was used to compare outcomes for EAPSI awardees and non-awardees. The 
primary threat to the validity of a quasi-experimental design comes from selection bias, namely, the 
possibility that pre-existing differences between awarded and unfunded applicants, rather than the 
EAPSI award itself, are responsible for observed differences in outcomes between the two groups. 

To reduce the risks associated with selection bias, the study incorporated propensity score analysis 
(PSA) to construct groups of awardees and non-awardees that were statistically similar across a 
number of pre-existing characteristics (i.e., gender, prior international experience, etc.). 34 
Subsequent impact models incorporated the results of the PSA. These methods are described in 
more detail in Appendix C. 

For each outcome, the impact of EAPSI was estimated for a matched group of awarded and 
unfunded applicants, controlling for number of years since EAPSI application, under-represented 
minority status, gender, and where applicable, the number of pre-award publications and field of 
study. Exhibits display the adjusted (estimated) means for awarded and unfunded applicants, the 
estimated impact, the standard error, and the p-value. For model specifications and standard error 
calculations, see Appendix C. 

It is important to note that the findings described here result from a quasi-experimental analysis 
that incorporates statistical controls. Although the propensity score analysis is used to control for 
pre-existing characteristics along which awarded and unfunded applicants may have differed, there 
is always a chance that some unmeasured preexisting characteristic, rather than the EAPSI award 
itself, could be responsible for any difference in outcomes between these two groups. 

2.4.3 Comparative Benchmarking 

For this study, the primary comparison group for EAPSI fellows is a propensity-score matched 
sample of unfunded EAPSI applicants. The Survey of Doctoral Recipients (SDR) and National Survey 


34 PSA is a common quasi-experimental design approach that has been shown to produce unbiased 
estimates of program effects. See for example Rosenbaum, P., & Rubin, D. 1984. Reducing bias in 
observational studies using subclassification on the propensity score. Journal of the American Statistical 
Association, 79(387), 516-524; Heckman, J., Ichimura, H., Smith, J., & Todd, P., Characterizing selection 
bias using experimental data, available from 

http://www.irp.wisc.edU/initiatives/trainedu/igrfp/readings04/Heckman_Characterizing_selection_bias.p 
df; Cook, T., Shadish, W. and Wong, V. 2008. Three conditions under which experiments and observational 
studies produce comparable causal estimates: New findings from within-study comparisons, Journal of 
Policy Analysis and Management, 27(4), 724-750. 
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of Recent College Graduates (NSRCG) respondents were used as a secondary comparison group to 
benchmark fellows' and all applicants' outcomes on specific indicators compared to national 
averages. The methods used for these analyses are described in more detail in Appendix D. 

Data from the 2006 and 2008 SDR were used for comparisons with: EAPSI fellows and all applicants 
whose highest degrees were a doctorate. Data from the 2006 and 2008 NSRCG were used for 
comparisons with EAPSI fellows and all applicants whose highest degrees were a master's. 
Comparisons were made for employment and international collaboration indicators. 


Abt Associates Inc. 


2. Methodology 20 


Evaluation of the East Asia and Pacific Summer Institutes Program: Final Report 


3 Characteristics of EAPSI Applicants and Hosts 


This chapter presents information on the EAPSI application process, trends in program applications 
and awards, and characteristics of awarded and unfunded applicants and hosts, as well as their 
motivations for applying for and participating in EAPSI. Data for this chapter come from NSF's 
program records and web surveys of applicants, hosts, and fellows' graduate advisors, as well as 
from interviews conducted with the EAPSI host site representatives— the agency officials and 
program managers most familiar with the EAPSI program in each of the host locations. 

Specifically, this chapter answers the following questions: 

• What are the characteristics of people who apply for and participate in the EAPSI program? 

• What motivates individuals to apply for and participate in the program? 


3.1 Key Findings 


• The EAPSI program awarded the fellowship to just over half (56 percent) of the applicants from 
2002 to 2009, totaling 1,126 fellowships. The number of applications received by the program in 
a given year exhibited an upward trend coinciding with the addition of new host sites to the 
program. 35 

• Applicants most commonly applied to EAPSI to enhance their skills or knowledge as a researcher 
(82 percent), to learn about the culture, history, and geography of another country (77 percent), 
and to conduct research with a specific person or at a specific institution (70 percent). 

• A large majority of graduate advisors reported that they encouraged their students to apply for 
the fellowship (90 percent). Of these, most viewed the program as one that would help their 
students' future academic or professional career (89 percent), and was important for them to 
gain an international perspective (88 percent). 

• More applicants were pursuing a doctoral degree than a master's degree at the time of 
application (76 versus 24 percent). The three most common fields of study for applicants were 
biological, agricultural, or environmental sciences (28 percent), engineering (27 percent), and 
physical sciences (21 percent). 

• Graduate advisors typically assist their students with the EAPSI application process. As an 
example, many fellows reported that they received a letter of recommendation from their 
advisor (87 percent). 

• Host researchers most commonly reported that they agreed to serve as a host in the program 
because of a shared interest in the prospective fellow's research project (64 percent), an 
interest in creating an international environment in their research group (59 percent), and to 
establish or maintain collaboration with a U.S. researcher (57 percent). 


Program information did not contain information on unfunded applicants for 2000 and 2001, so these 
years were not included in this piece of the analysis. 
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• The EAPSI program tends to award fellowships to students who would be establishing new 
relationships with their host researcher. Only 15 percent of hosts reported that they knew their 
prospective fellow prior to him/her applying. Almost half (45 percent) of the hosts, however, 
knew the graduate advisor of the prospective fellow. 


3.2 Applications and Awards 


3.2.1 Trends in Program Applications and Awards 

According to NSF's program data, the EAPSI program received 2,021 completed applications from 
2002 to 2009. 36 Because EAPSI is funded and managed in partnership with the participating 
organizations or agencies in the host countries, the final slate of selected participants is agreed to by 
both NSF and the foreign agencies. Exhibit 3.1 shows, for the years from 2002 to 2009, the number 
of applications received, the number of fellowships awarded, and the resulting percent of 
applications that were successful. 

The number of applications received in a single year averaged 253 per year, ranging from 99 to 503. 
The increasing number of applications received coincides with the addition of new host countries to 
the EAPSI program. In 2002 and 2003, when the participating host sites were only Japan, Korea, and 
Taiwan, the average annual number of applications received was 88. Australia and China established 
EAPSI programs in 2004, bringing the number of locations to five, and the average number of 
applications grew in the next two years (to 226 in 2004, 234 in 2005, and 217 in 2006). Further 
increases in applicants were seen in 2007 when New Zealand joined the program and again in 2008 
when Singapore joined. The number of applications for 2003 is likely a subset of the total number of 
applications received because the program data available for 2003 was incomplete in the program 
files available to the evaluation team. 

From 2002 to 2009, 56 percent of applicants to EAPSI were awarded the fellowship, for a total of 
1,126 fellowships; the annual award rate ranged between 37 and 85 percent. The award rate in any 
given year depends on the number of applications as well as the number of fellowships available in a 
given year, which varies based on the number of participating host countries, level of funding for the 
program, and the number of fellowship slots per host site in a given year. Applications from 2000 
and 2001 are not included in these analyses because program records for these years were limited 
to only the fellows, and did not include records of unfunded applicants. 


Although individuals who applied in 2000 and 2001 are included in the study's sampling frame, NSF's 
program data for this time period were limited to only the fellows, so these two years have been excluded 
from analyses of trends in program applications and awards. The number of applications received in 2003 
and 2009 are likely to contain only a subset of the total number of applications received in those years; 
program data available for that year appears to be incomplete at the time data were collected for the 
study. 
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Exhibit 3.1: EAPSI Applications Received and Awarded, 2002-2009 



♦ Percent Awarded B Number Awarded Number of Applications Received 

2002 - 2003: Japan, Korea and Taiwan 

2004 - 2006: Japan, Korea, Taiwan, and Australia and China 

2007: Japan, Korea, Taiwan, Australia, China and New Zealand 

2008: Japan, Korea, Taiwan, Australia, China, New Zealand and Singapore 

NOTES: Data from 2000 and 2001 are not included in these analyses because program data for these years was limited to 
only the fellows, and did not include records of unfunded applicants. Numbers for 2003 are likely understated because 
program records were incomplete in the files available for the evaluation 
SOURCE: NSF Extant Program Data 

3.2.2 Applicants’ Reasons for Applying 

Applicants' stated reasons for applying to the fellowship offer insight into the elements of the EAPSI 
program that were most appealing to graduate students (Exhibit 3.2). Most commonly, the 
applicants wanted to enhance their skills or knowledge as a researcher (82 percent), to learn about 
the culture, history, and geography of another country (77 percent), and to conduct research with a 
specific person or at a specific institution (70 percent). For the most part, awarded and unfunded 
applicants had similar motivations for applying; however, a few differences are worth noting. 
Awarded applicants' reasons for applying more often included to learn about culture, history and/or 
geography of another place (82 versus 66 percent) and to learn another language (30 versus 14 
percent) than unfunded applicants. 
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Exhibit 3.2: Reasons for Applying 


Awarded 


To enhance my skills or knowledge as a researcher 81.6% 81.0% 82.7% 


To learn about the culture, history, and/or 
geography of another place 

76.5 

81.5 66.3 

To conduct research with a specific person or at a 
specific institution 

69.9 

68.9 

72.1 

To collaborate with a foreign scientist 

68.2 

69.4 

65.6 

To travel outside of the U.S. 

66.5 

67.6 

64.2 

To understand what research in my field was like 
outside the U.S. 

65.1 

65.3 

64.7 

To enhance my resume as a future job candidate 

60.1 

60.1 

60.1 

To make progress towards earning my graduate 
degree 

35.7 

35.9 

35.3 

To access resources for research that 1 could not 
find in the U.S. 

29.9 

29.9 

29.7 

To learn another language 

24.5 

29.5 

14.1 

For family reasons 

2.6 

2.6 

2.6 

Other reasons not listed above 

3.6 

3.2 

4.2 


NOTES: N=1302 (938 Awarded, 364 Unfunded Applicants), Missing=l (Unfunded Applicant). Responses do not sum to 
100 because multiple responses were permitted. 

SOURCE: EAPSI Applicant Survey-Item B1 


Unfunded 


All Applicants 


Graduate advisors of awarded applicants provided 
additional insight into the perceived value of the EAPSI 
program. Ninety percent of the advisors who 
responded to the survey reported that they 
encouraged their students to apply for the fellowship. 37 
When asked why, most believed it would help their 
students' future academic or professional career (89 
percent) and that it was important for them to gain an 
international perspective (88 percent). More than half 


This program provides unique 
opportunities to further scientific 
and global perspectives. It has huge 
academic and cultural value to any 
student who can find academic 
colleagues in the appropriate 
countries. (Advisor of EAPSI fellow) 


The department has always encouraged 
international collaboration, even before the 
EAPSI program. However, this program 
reinforced the department's goals and 
encouraged other graduate students to 
explore international collaboration (Advisor 
of EAPSI fellow) 


of the advisors also encouraged their students 
to participate believing it would foster more 
international collaborations between 
researchers in the U.S. and abroad (69 percent) 
and because of their own positive experiences 
collaborating internationally (68 percent). 38 

A notable percentage of advisors, however, 
could not recall if their institution publicized the 


No exhibit. N=641, Missing=3 Advisors. Source: EAPSI Advisor Survey-Item B5. 

No exhibit. N=579, Missing=0 Advisors. This item was only answered by advisors whose response to B5=l 
(indicated that they encouraged fellow to apply to EAPSI program). Responses do not sum to 100 because 
multiple responses were permitted 
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program in the various ways listed in Exhibit 3.3. However, some of the fellow advisors reported 
that their institutions provided application information through departments (32 percent) and 
actively encouraged individual graduate students to apply to the program (27 percent). 


Exhibit 3.3: Institutional Efforts to Publicize EAPSI Program as Reported by Advisors 


Provided application information about EAPSI in 
departmental activities 

Actively encouraged individual graduate students 
to apply for the EAPSI program 

Posted EAPSI information on departmental 
bulletin boards 

Former EAPSI fellows presented their EAPSI 
experiences to other graduate students 

Maintained current EAPSI program information at 
the student research/internship/or career center 

Held on-campus meetings about EAPSI 
Other ways not listed above 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
■ Yes ■ No ■ Do not recall 


NOTES: N=608 to 641, Missing 3 to 36 Advisors. 
SOURCE: EAPSI Advisor Survey- Item B2 


1111 



Advisors also provided information that characterized the applicants' home departments' attitudes 
towards international collaboration at the time they enrolled in EAPSI (Exhibit 3.4). Roughly three- 
quarters of advisors characterized the department as one that encouraged international 
collaborations (74 percent) and had hosted foreign researchers (73 percent). However, fewer 
described the department as promoting international fellowship research opportunities (37 percent) 
or rewarding or funding faculty for their international collaborations (22 percent). 


Exhibit 3.4: Characteristics of Fellows' Graduate Department as Reported by Advisors 



Advisors 

Department encouraged international collaborations 

74.0% 

Department hosted foreign post-docs and faculty visiting the institution for research- 
related purposes 

72.9 

Department promoted (i.e., advertised) fellowships for graduate students or 
postdocs to conduct research in a foreign country 

36.8 

Faculty in this department were rewarded for developing international collaborations 

22.4 

Department provided financial support to faculty pursuing international 
collaborations 

9.7 

Do not recall 

3.0 

None of the above 

9.1 


NOTES: N=639, Missing=5 Advisors. 
SOURCE: EAPSI Advisor Survey-Item El 
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3.3 Characteristics of Applicants 


The EAPSI program targets U.S. citizens or permanent residents who are enrolled in research 
oriented master's or PhD degree programs, in science and engineering research and education fields 
supported by NSF. As such, EAPSI applicants represent a variety of demographic and disciplinary 
backgrounds and come from institutions across the U.S. 

3.3.1 Applicants’ Demographic Characteristics 

Demographic characteristics of awarded and unfunded applicants were similar (Exhibit 3.5). In both 
groups, more individuals were male (55 percent) than female (45 percent). The majority of 
respondents identified their race as White (75 percent). The remaining respondents self-identified 
as Asian (17 percent), multi-racial (4 percent), Black or African American (3 percent), American 
Indian or Alaskan Native (less than 1 percent), or Native Hawaiian or Pacific Islander (less than 1 
percent). 


Exhibit 3.5: Applicants' Demographic Characteristics 



All Applicants 

Awarded 

Unfunded 1 

Gender 

Female 

44.9% 

43.7% 

47.4% 

Male 

55.1 

56.3 

52.6 

Race 

White 

74.6% 

74.4% 

75.2% 

Asian 

17.1 

17.9 

15.3 

Multiracial (two or more races) 

4.3 

4.8 

3.4 

Black or African American 

3.3 

2.3 

5.2 

Native Hawaiian or other Pacific Islander 

0.4 

0.2 

0.9 

American Indian or Alaska Native 

0.3 

0.5 

0.0 

Ethnicity 

Non-Hispanic 

94.8% 

96.1% 

92.3% 

Hispanic 

5.2 

3.9 

7.7 

Disability Status 

Disabled 

1.4% 

0.8% 

2.5% 

NOTES: All Applicants: N=1243 to 1303, Missing: 

=0 to 60; Awarded Applicants: N= 

=900 to 938, Missing= 

=0 to 38; 


Unfunded Applicants: N=343 to 365, Missing=0 to 22. 

SOURCE: EAPSI Applicant Survey-Items Gl-G5a, NSF Extant Data 


3.3.2 Applicants’ Academic Characteristics 

Although the EAPSI program provides fellowships to graduate students at both master's and 
doctoral levels, applicants are more likely to be doctoral students (Exhibit 3.6); three times as many 
applicants were pursuing doctoral degrees than master's degrees at the time they applied for the 
fellowship (76 versus 24 percent). The fellows were more likely to be pursuing doctoral degrees than 
unfunded applicants (79 versus 69 percent). 

The EAPSI program invites applications from a range of science and engineering fields. The three 
most common disciplines of study at the time of application were: biological, agricultural, or 
environmental sciences (28 percent), engineering (27 percent), and physical sciences (21 percent) 
(Exhibit 3.6). 
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Exhibit 3.6: Degree Type and Discipline of Study at Time of Application 



All Applicants 

Awarded 

Unfunded | 

Type of Degree Pursued a 

PhD or equivalent 

75.9% 

79.2% 

68.9% 

Masters 

24.2 

20.8 

31.1 

Discipline of Study 

Biological, agricultural, or environmental sciences 

28.1% 

27.1% 

30.0% 

Engineering 

26.6 

27.4 

25.0 

Physical sciences 

20.5 

21.8 

17.9 

Computer and information sciences, mathematics, 
or statistics 

9.7 

10.3 

8.6 

Psychology and social sciences 

9.7 

7.8 

13.5 

Health 

5.4 

5.6 

5.1 


a N=1301 (938 Awarded, 363 Unfunded Applicants), Missing=2 Unfunded Applicants.. PhD or equivalent includes EdD, 
MD, joint MD/PhD, PsyD, and ScD. 

b N=1303 (938 Awarded, 365 Unfunded Applicants), Missing=0. 

SOURCE: EAPSI Applicant Survey-Items A5 and A7 

EAPSI applicants tended to have strong academic backgrounds. The large majority of respondents 
(84percent) reported a GPA of 3.25 to 4.00, equivalent to a B+ or higher (Exhibit 3.7). 


Exhibit 3.7: Self-Reported Undergraduate GPA 


50 


40 


« 30 

c 

01 

U 

i- 

0) 

20 


10 


43.0 43 -9 



- 16 S- 


13.3 


1.6 


2.4 2.3 2.8 


3.75 -4.00 3.25 -3.74 2.75- 3.24 2.74 or below 

Self-Reported Undergraduate GPA 

I All Applicants (N=1289) ■ Awarded (N=928) ■ Unfunded (N=361) 


NOTES: N=1289 (928 Awarded, 361 Unfunded Applicants), Missing=14 (10 Awarded, 4 Unfunded Applicants. Missing 
includes 7 responses of "Did not receive grades." 

SOURCE: EAPSI Applicant Survey-Item Cl 


Further, prior to applying to EAPSI, 23 percent of applicants had received a nationally competitive 
fellowship to support their graduate studies; more awarded than unfunded applicants had 
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fellowships (26 versus 18 percent). 39 EAPSI applicants also reported an average of 3.7 publications 
which included peer-reviewed conference papers, peer-reviewed journal articles and book chapters 
at the time of application; 8 percent of these included a foreign collaborator (Exhibit 3.8). Additional 
ways in which EAPSI applicants engaged in international collaborations are discussed in the next 
section. 


Exhibit 3.8. EAPSI Applicants' Publications and Patents at Time of Application 



All Applicants 

Awarded 

Unfunded | 

Mean number produced 

Mean N 

Mean N 

Mean N 

Publications 

3.7 

3.7 

3.8 

Patents 

0.1 

0.1 

0.1 

Percent produced with a foreign collaborator 

Mean % 

Mean % 

Mean % 

Mean percent of publications 

8.3 

8.7 

7.2 

Mean percent of patents 

0.4 

0.6 

0 


NOTES: All Applicants: N=1115 to 1136, Missing=188 to 167 ; Awarded Applicants: N=811 to 830, Missing=126 to 108; 
Unfunded Applicants: N=306 to 303, Missing=62 to 59. 

SOURCES: EAPSI Applicant Survey-Item C8 


3.4 Application Decisions and Support 


Application materials are typically due in November or December. In addition to an application 
form, applicants are to include a biographical sketch, a project description that addresses the 
intellectual merit and broader impacts of their proposed research and expected benefits of working 
with the host, letters of reference, and undergraduate and graduate transcripts. 

3.4.1 Selection of Host Researchers / Host Institutions 

The EAPSI program gives students the opportunity to identify a specific host researcher and 
institution of their choice within a wide variety of institutions and geographic locations. Applicants 
may approach individuals at almost any academic or research institution in one of the participating 
EAPSI locations to seek acceptance and placement. 

Applicants must specify at least one potential host location and corresponding host researcher and 
institution, but may include additional choices in order of preference. In 2008, an application 
requirement was added whereby applicants must obtain a preliminary invitation or acceptance from 
their top choice host researcher and include that correspondence with their application documents 
(prior to 2008, applicants were strongly encouraged to submit this type of correspondence). This 
correspondence is intended to provide evidence of the host's willingness to work with the applicant 
and a mutual commitment to the proposed research project. Over half of the applicants (65 
percent) reported that they submitted evidence of support (e.g., a letter or an email message) from 
their proposed host scientist. A higher proportion of awarded than unfunded applicants reported 


No exhibit. N=1286 (929 Awarded, 357 Unfunded Applicants), Missing=17 (9 Awarded, 8 Unfunded 
Applicants). Nationally competitive fellowships were defined as unrestricted fellowships granted by a 
federal agency, private foundation, or similar organization directly to an individual graduate student (or 
graduate school applicant) for use at any graduate institution of his/her choosing. Source: EAPSI Applicant 
Survey-Item C9. 
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that they provided this type of letter or other communications (68 versus 58 percent) when 
applying; 23 percent of all applicants could not recall if their application materials included any 
correspondence with their proposed host. 40 

Hosts of awarded applicants were asked about the ease of providing the supporting letter for their 
prospective fellow's application to NSF and a majority of respondents indicated that it was easy (74 
percent); just 10 percent reported that it was somewhat or very difficult to provide the materials, 
and 16 percent responded that they did not provide these materials. 41 

3.4.2 Role of Advisors in Application 

As graduate students, EAPSI applicants are still developing as independent researchers and their 
mentors are likely to play an important role in the opportunities they choose to pursue. Only 2 
percent of respondents reported that their advisor opposed their decision to apply. Indeed, advisors 
and other mentors played important roles in the EAPSI application (no exhibit). 

A large proportion of applicants (87 percent) verified that their graduate advisor or other mentor 
provided a letter of recommendation, although awarded applicants were slightly more likely than 
unfunded applicants to report such support (90 versus 81 percent, Exhibit 3.9). Just over half of the 
applicants (53 percent) indicated that their advisor or mentor provided feedback on their research 
project proposal. Further, in preparing their EAPSI 


40 No exhibit. N=1296 (935 Awarded, 361 Unfunded Applicants), Missing=7 (3 Awarded, 4 Unfunded 
Applicants). Source: EAPSI Applicant Survey-Item C7a. 

41 No exhibit. N=657, Missing=8 Hosts. This item was only answered by hosts whose response to A2=l 
(indicated that they did host the EAPSI graduate student named in the survey). Source: EAPSI Host Survey- 
Item B7. 


application, applicants indicated that their advisor or 
mentor recommended them to a colleague at the host 
institution (40 percent) and suggested a potential host 
institution (37 percent). Awarded applicants were more 
likely than unfunded applicants to receive this type of 
assistance from their advisor or mentor. Specifically, 44 
percent of awarded versus 31 percent of unfunded 
applicants were recommended to a colleague at the host 
institution and 42 percent versus 28 percent, respectively, 
received suggestions for a host institution from their 
advisor. 




opportunity for graduate 
students to obtain experience 
(both research experience and 
other experiences abroad). At 
its most effective [sic], it adds 
a significant new dimension to 
the dissertation research of 
participating students. 

(Advisor of EAPSI fellow) 
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Exhibit 3.9: Support Received by Applicants from Advisor or Other Mentor 



All Applicants 
(N=1302) 

Awarded 

(N=938) 

Unfunded jj 
(N=363) § 

Provided a letter of recommendation to NSF 

87.2% 

90.1% 

81.2% 

Provided feedback on my project proposal 

52.9 

53.3 

52.0 

Recommended me to a colleague at the host institution 

39.6 

43.8 

30.7 

Suggested a host institution 

37.2 

41.5 

28.2 

Discussed language and cultural aspects of the host site 

12.0 

15.4 

5.0 

Other reasons not listed above 

2.3 

2.5 

1.8 

None 

6.2 

4.4 

10.0 


NOTES: N=1302 (938 Awarded, 364 Unfunded Applicants), Missing=l Unfunded Applicant. Responses do not sum to 
100 because multiple responses were permitted. 

SOURCE: EAPSI Applicant Survey-Item B5 


Graduate advisors' responses were consistent with 
those of the fellows. 42 A majority of the advisors 
recalled providing a letter of recommendation to NSF 
(89 percent), letter of recommendation to the host 
researcher (70 percent), and general application 
preparation support (68 percent). Further, advisors 
confirmed supporting their graduate students in 
selecting the host researcher or institution for the 
fellowship: 65 percent suggested a researcher or 
institution and 53 percent helped contacting the 
hosts (Exhibit 3.10). 


It is a great opportunity for students to 
learn new techniques, develop new 
collaborations, learn about science in a 
different country, and experience life in 
a different culture— all wonderful 
aspects of the program! (Advisor of 
EAPSI fellow) 


Exhibit 3.10: Support Provided by Graduate Advisors 


Percent of Advisors 

Provided a letter of recommendation to NSF 

88.8 

Provided a letter of recommendation to EAPSI host 

70.1 

Provided support with application preparation 

68.0 

Located host institution or host researcher 

65.2 

Contacted host institution 

53.4 

Other 

1.2 


NOTES: N=618 to 640, Missing=4 to 26 Advisors. Responses do not sum to 100 because multiple responses were 
permitted. 

SOURCE: EAPSI Advisor Survey-Item B3 


The graduate advisors surveyed for this study were limited to those of awarded applicants (fellows), thus 
their responses do not represent advisors of unfunded applicants. 
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3.5 Applicants’ Prior International Experiences and Relationships 


EAPSI offers international opportunities that most applicants have not previously experienced. This 
section describes the prior international experiences of participants. 

3.5.1 Prior Academic and Travel Experiences Outside the U.S. 

The majority of EAPSI applicants did not have extensive academic and travel experiences outside the 
U.S. at the time of application (Exhibit 3.11). Only 36 percent of applicants participated in study 
abroad programs as undergraduates or graduate students, and 18 percent had attended elementary 
or secondary school outside the U.S. Around one-third had lived outside the U.S. for six months or 
longer (35 percent) or visited or lived in one of the seven EAPSI host locations for a month or longer 
(30 percent). Less than a quarter of applicants (22 percent) reported that they had attended a 
research conference outside the U.S. 


Exhibit 3.11: Prior Academic and Travel Experiences Outside the U.S. 



All 




Applicants 

Awarded 

Unfunded 

Prior study abroad experience (semester or 
longer) 

36.1% 

34.7% 

39.2% 

Lived outside the U. S. (6 months or longer) 

35.0 

34.2 

36.8 

Visited or lived in an EAPSI host location (one 
month or longer) c 

29.6 

28.0 

33.0 

Attended or presented scholarly work at a 
research conference outside the U.S. 

22.0 

21.1 

24.0 

Attended elementary or secondary school 
outside the U.S. 

17.6 

16.5 

19.8 


NOTES: All Applicants: N=1294 to 1301, Missing=2 to 5 ; Awarded Applicants: N=935 to 937, Missing=l to 3; 

Unfunded Applicants: N=359 to 364, Missing=l to 6 .Percentages do not sum to 100 because multiple 
responses were permitted. 

SOURCE: EAPSI Applicant Survey-Items C2a, C2b, C3 and C6. 

3.5.2 Prior Relationships with Individuals and Institutions Based Outside the U.S. 

One of the primary goals of EAPSI is to initiate professional relationships that might seed future 
productive collaborations. Few applicants had these professional relationships prior to application 
to the program (Exhibit 3.12). Just over one-quarter had collaborated on research with someone 
based outside the U.S. (27 percent), and fewer had participated on a research team with a visiting 
foreign scientist (19 percent), or published a research paper or book with someone based outside 
the U.S (9 percent). Awarded applicants were slightly more likely than unfunded applicants to work 
on a project with a visiting foreign scientist (21 versus 15 percent). Furthermore, most of the 
applicant respondents (86 percent) had no prior experience collaborating with their proposed host 
scientist or someone else at the host institution to which they applied. 

About one-third of applications appear to build on an existing collaboration between the 
prospective host and a faculty member at the applicants' U.S. institution (34 percent). This type of 
pre-existing collaboration was more likely to exist for applicants who received EAPSI awards than for 
unfunded applicants (38 versus 25 percent). The fellows were also more likely to know another 
graduate student who had been to the host institution (12 versus 7 percent). 
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Exhibit 3.12: Prior Relationships with Individuals and Institutions Outside the U.S. 


All Applicants 


Awarded 


Prior research experiences with foreign scientists 

Collaborated on research with someone based in a 
country outside the U.S. 

Participated on a research team with a visiting 
foreign scientist 

Published research with someone based in a 
country outside the U.S. 

Prior collaboration with proposed host scientist or someone else at proposed institution 


27.0% 


18.7 


8.9 


27.0% 


20.7 


9.1 


Unfunded 


26.8% 

14.6 

8.4 


No prior collaboration 

86.2% 

85.3% 

88.2% 

Collaborated with proposed host scientist 

11.6 

12.9 

8.7 

Collaborated with someone else at proposed host 
institution 

2.2 

1.8 

3.1 

Prior connection between graduate institution and proposed host institution 

Faculty member was / had collaborated with 
someone at the host institution 

33.9% 

38.3% 

24.6% 

Worked with a colleague who had completed an 
EAPSI fellowship 

13.0 

13.7 

11.8 

Other graduate students had been to the host 
institution 

10.0 

11.6 

6.9 

University had an existing collaboration or exchange 
program with the host institution 

5.8 

5.5 

6.4 


NOTES: All Applicants: N=1290-1300, Missing=3-13 ; Awarded Applicants: N=931-937, Missing=l-7 ; Unfunded 
Applicants: N=359-363, Missing=2-6 . Percentages do not sum to 100 because multiple responses were permitted. 
SOURCE: EAPSI Applicant Survey-Items C6, C7b, C7c 


Responses from graduate advisors confirmed that some fellowships built on existing relationships. 
Among the advisors who responded to the survey, 36 percent reported having a prior collaboration 
with their student's EAPSI host, 14 percent had collaborated with someone other than the EAPSI 
host at the host institution, and 11 percent had previously visited the student's host institution for 
professional purposes. 43 


3.6 Hosts 


3.6.1 Eligibility 

Information from the interviews with EAPSI agency officials and program managers in the foreign 
locations suggested that each host site sets its own requirements for host researchers and 
institutions. In general, eligibility to serve as a host and host institution is fairly broad. China and 
Singapore tend to have more strict eligibility requirements than Australia and New Zealand. Hosts 
who are willing to accept an EAPSI fellow are advised to get to know the prospective fellow during 
the application process. This involves reviewing and discussing their proposed research project. 
Since 2008, hosts are required to provide the prospective fellow with a letter of acceptance stating 
their intent and ability to serve as a host and agree to provide the fellow with any necessary 
laboratory facilities and supplies. 


No exhibit. N=644, Missing=0 Advisors. Source: EAPSI Advisor Survey-Item B4. 
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3.6.2 Reasons for Hosting an EAPSI Fellow 

In most locations, hosts are not provided with funding to host 
an EAPSI fellow, thus, their decision to serve as a host is driven 
by other factors (Exhibit 3.13). The most frequently reported 
reason for hosting an EAPSI fellow was a shared interest in the 
research project (64 percent). More than half of the 
respondents wanted to create an international environment in 
their research group (59 percent) and to establish/maintain 
collaboration with a U.S. researcher (57 percent). Half reported 
that they decided to serve as a host because they personally 
knew, knew of, or had previously collaborated with the fellow's 
graduate advisor, and 13 percent because they knew the student 


Exhibit 3.13: Reasons for Hosting an EAPSI Fellow 



Hosts 

(N=665) 

1 was interested in the project proposed by the fellow 

63.5% 

To create an international environment in my research group 

59.2 

1 was interested in establishing or maintaining collaboration with a U.S. 
researcher 

56.8 

1 personally knew, knew of, or previously collaborated with the fellow's graduate 
advisor 

50.4 

My research area is particularly suitable for international collaboration 

36.4 

To attract new students/postdocs to my research 

32.6 

1 had a positive experience with another international program 

28.1 

1 personally knew, knew of, or previously collaborated with researchers at the 
fellow's institution 

19.1 


To improve the status of my department and/or institution 
To learn new methodologies, approaches, or tools from the fellow 
I had a positive experience with the EAPSI program in the past 
I personally knew, knew of, or previously collaborated with the fellow 
In my field, individuals trained at U.S. graduate institutions are highly sought 
after 

Other reasons not listed above 3.2 

NOTES: N=665, Missing=0 Hosts.). This item was only answered by hosts whose response to A2=l (indicated that they 
did host the EAPSI graduate student named in the survey). Responses do not sum to 100 because multiple responses 
were permitted. 

SOURCE: Host Survey-Item B2 

Thirty percent of the hosts who responded to the survey did not know the fellow or the fellow's 
advisor prior to hosting the fellow. Only 15 percent of the hosts who responded to the survey stated 
that they formerly knew the fellow, slightly less (11 percent) knew both the fellow and the fellow's 
graduate advisor, yet almost half (45 percent) reported that they knew the fellow's graduate advisor 
but not the fellow beforehand. 44 Overall, hosts and awarded applicants had similar responses in 


16.4 
14.7 
14.6 

13.4 

12.3 


The program deepens 
international friendship, 
particuiariy between young 
researchers. It is one of the 
rare occasions when 
graduate students can 
interact with foreign 
researchers on daily basis. 


No exhibit. N=659, Missing=6 Hosts. This item was only answered by hosts whose response to A2=l 
(indicated that they did host the EAPSI graduate student named in the survey). Source: EAPSI Host Survey- 
Item B6. 
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regards to a pre-existing relationship, which indicates that the EAPSI program tended to award 
fellowships to students who would be establishing new relationships with their host researcher. 

3.6.3 Concerns about Hosting an EAPSI Fellow 

Hosts were asked about any concerns they had with regards to hosting their EAPSI fellow (Exhibit 
3.14). Roughly a third of the respondents did not have any. The most common concern, reported by 
54 percent of host respondents, was that the length of EAPSI (8 to 10 weeks) was too short for their 
fellow's proposed project (Exhibit 3.13). 


Exhibit 3.14: Host Concerns about Hosting an EAPSI Fellow 



Hosts 

(N=665) 

The length of EAPSI was too short for the proposed project 

54.4% 

1 was concerned about the fellow's commitment 

11.0 

1 was concerned about integrating this fellow into my group 

8.9 

1 was concerned that 1 might not benefit from hosting this student 

5.0 

In my field, individuals trained at U.S. institutions sometimes have gaps in their 
knowledge, skills, or abilities 

2.4 

1 was concerned about the risks of international collaboration in general 

1.8 

The fellow's proposed research project was especially risky 

1.5 

1 had a negative experience with other graduate student fellows (not affiliated 
with EAPSI) 

0.8 

1 had a negative experience with EAPSI in the past 

0.5 

My research area is not particularly suitable for international collaboration 

0.5 

Other concerns not listed above 

2.6 

No concerns selected a 

35.3 


NOTES: N=665, Missing=0 Hosts. This item was only answered by hosts whose response to A2=l (indicated that they did 
host the EAPSI graduate student named in the survey). Responses do not sum to 100 because multiple responses were 
permitted. 

a These include those who did not select any concern, and those who wrote in "none" or "no concerns" in the other 
option. 

SOURCE: Host Survey-Item B3 

3.6.4 Prior Academic and Travel Experiences in the U.S. 

Nearly all the hosts who responded to the survey (97 percent) indicated that they had visited the 
U.S. for educational, research or other professional purposes prior to serving as an EAPSI host 
(Exhibit 3.15). The most frequently cited reason was to attend a conference, workshop, or meeting 
(74 percent). Half of the hosts had been visiting scientists in the U.S and over one-quarter were 
postdoctoral fellows (29 percent) and/or attended graduate school (28 percent) in the U.S. 
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Exhibit 3.15: Host Reasons for Travel to the U.S. 


Hosts 

(N=665) 


Attended a conference, workshop or meeting 74.0% 


Served as a visiting scientist 

49.9 

Held a postdoctoral fellowship 

28.9 

Attended graduate school 

27.8 

Held a faculty position 

8.7 

Attended an undergraduate institution 

4.4 

Other reasons not listed above 

5.1 

Did not visit the U.S. for any professional purposes 

2.7 


NOTES: N=665, Missing=0 Hosts. This item was only answered by hosts whose response to A2=l (indicated that they did 
host the EAPSI graduate student named in the survey). Responses do not sum to 100 because multiple responses were 
permitted. 

SOURCE: Host Survey-Item B5 
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4 Fellowship Activities 


This chapter describes the EAPSI fellowship from 
the perspectives of EAPSI fellows, their hosts, the 
fellows' U.S. advisors, and the managers and 
officials in the host locations. 

Data for this chapter come from extant NSF 
program records; web surveys of fellows, hosts, 
and graduate advisors of fellows; and interviews 
with EAPSI representatives from each of the seven 
host sites. Statistically significant differences 
observed between the sites are noted. 

Specifically, this chapter addresses the following 
question: 

• What are the program experiences of 
program participants? 


It is an incredible, invaluable 
experience. Science is getting more 
international all of the time and it is 
critical that we understand how it is 
conducted in other countries/cultures. 
It also helps one really become more 
understanding of one's foreign 
colleagues when working at U.S. 
institutions. It's an educational, 
incredibly fun adventure!!! What is 
there not to recommend? (EAPSI 
fellow) 


4.1 Key Findings 


• Various types of support were available to fellows as they transitioned into the research 
activities in their host site. Fellows reported attending a pre-departure orientation facilitated by 
NSF (87 percent) and host researchers indicated that they assisted their fellow with getting 
access to needed facilities (90 percent), arranging their office space (83 percent), and finding 
them an appropriate research project (75 percent). 

• A majority of hosts agreed that that their research interests were well-matched with those of 
their fellow (82 percent), and that they and the fellow shared similar goals and expectations for 
the EAPSI experience (60 percent). 

• Fellows had the opportunity to work with a range of individuals at the host institution. Flosts 
most frequently reported that their fellow worked with the host (80 percent), graduate students 
in the host's research group (66 percent), and other scientists in the host's research group (45 
percent). 

• In general, a larger proportion of fellows reported that they worked on research activities 
independently as opposed to equally with their host. Activities most commonly cited by fellows 
as being worked on equally were interpreting results (37 percent), collecting data or carrying out 
simulations (33 percent), developing instrumentation (31 percent), and developing ideas or 
hypotheses (31 percent). 

• Overall, fellows were satisfied with their EAPSI research activities and interactions with their 
host. A majority of fellows reported that they were very satisfied with their host's expertise (81 
percent) and the match between their research interests and the host's (63 percent). 
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• Hosts provided positive reports of their interactions with the fellows. At least 85 percent 
strongly agreed or agreed that their fellow demonstrated sufficient knowledge and expertise, 
integrated well with their research group, was hard-working and dedicated, and spent sufficient 
time working on the project. 

• A minority of fellows noted difficulties related to their research activities and host interactions. 
Among the difficulties cited were not enough guidance from the host (17 percent), and being 
assigned a role that was less than merited by their skills or knowledge (5 percent). Less than 10 
percent of hosts reported specific challenges regarding collaborating on research with their 
fellow. 

• While conducting research in the host site is a primary focus of the program, fellows are 
encouraged to engage in other professional activities. Fellows most often reported that they 
visited other institutions (72 percent), networked with colleagues not affiliated with their host 
institution (64 percent), and gave a presentation or talk (63 percent). Close to 90 percent were 
satisfied or somewhat satisfied with the professional connections made during their fellowship. 

• Nearly all fellows participated in cultural and leisure activities in their host site which included 
sightseeing (97 percent), exploring the landscape or geography (91 percent), and visiting 
museums (89 percent). 

• Communication or language issues were most frequently cited as difficulties that fellows 
encountered during their fellowship; however, this varied significantly by host site where a 
greater number of fellows in primarily non-English speaking host sites experienced these 
difficulties (28 to 44 percent within these sites) than fellows in primarily English-speaking host 
sites— New Zealand, Australia, and Singapore (0 to 10 percent within these sites). 

• Host site representatives indicated that they were generally satisfied with the level of 
communication and involvement of NSF in administering the program. Although few suggestions 
for improving the role of NSF were reported, the most common suggestion was for NSF to 
establish a reciprocal program whereby students could have opportunities to conduct research 
in the U.S. alongside American hosts. 


.2 Host Site Placement 


NSF and the foreign co-sponsoring organizations work together to finalize host placements and 
subsequent arrangements for fellows. When applying, applicants are advised that NSF may choose 
to award them a fellowship, but at a host site other than what they indicated as preferred in their 
application. Most fellows (95 percent) reported that they received their first choice of host site, 
although this varied by preferred site, from 100 percent for those who selected Singapore to 79 
percent among those who selected Australia. 45 

The distribution of fellows from 2000 to 2009 among the seven host sites reflects, in part, the 
number of years a site has participated in the EAPSI program and the number of fellowship slots per 


No exhibit. N=938, Missing=0 (fellows). Source: EAPSI Applicant Survey-Item A3 
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site (Exhibit 4.1). For example, Singapore, the newest site to establish an EAPSI program, was host to 
only 3 percent of fellow respondents. Taken as a whole, EAPSI fellows from 2000-2009 were more 
likely to carry out their fellowship in Japan (48 percent), followed by China (15 percent), and Taiwan 
(10 percent), the longest participating sites (Exhibit 4.1). Countries joined the EAPSI program at 
different times during the 2000-2009 period covered by this evaluation. Japan, Korea, and Taiwan 
were part of the program in 2000. Australia and China joined in 2004, New Zealand in 2007, and 
Singapore in 2008. Exhibit 4.1 displays the host site placements during these different periods. 


Exhibit 4.1: EAPSI Fellows' Placement Site for Active Host Sites from 2000-2009 



2000-2003 

2004-2006 

2007 

2008-09 

Overall 

Host Site 

% N 

% N 

% N 

% N 

% N 


36.3% 41 32.8% 100 48.3% 449 

9.7 11 7.2 22 9.5 88 

8.0 9 11.5 35 10.4 97 

13.3 15 10.8 33 9.9 92 

22.1 25 22.0 67 15.4 143 

10.6 12 7.2 22 3.7 34 

8.5 26 2.8 26 

100 113 100 305 100 929 

NOTES: Missing=14 Awarded Applicants. Japan, Korea, and Taiwan were the only participating host sites between 2000 
and 2003. Australia and China established programs in 2004, followed by New Zealand in 2007 and Singapore in 2008. 
SOURCES: NSF Extant Program Data and EAPSI Applicant Survey-Item A3 


Japan 

79.8% 

178 

45.1% 

130% 

Korea 

9.0 

20 

12.2 

35 

Taiwan 

11.2 

25 

9.7 

28 

Australia 



j 15.3 

44 

China 



17.7 

51 


New Zealand 
Singapore 

Total 100 223 100 288 



4.3 Foreign Language Preparation 


Fellows are not required to know the language of the host country in order to participate in EAPSI, 
although NSF strongly encourages them to obtain some language training prior to the fellowship 
period. New Zealand, Australia, and Singapore are primarily English-speaking countries and of the 
fellows who were placed in those sites, few reported that they engaged in studying the language of 
their host site to prepare for the fellowship (5 percent). Not surprising is that a much higher 
proportion of fellows placed in primarily non-English speaking host sites— Japan, Korea, Taiwan, and 
China— reported doing some type of language training in preparation for their international 
fellowship experience (67 percent). This included self-guided study (47 percent), a formal language 
training course (23 percent), and working with a conversational partner or tutor (9 percent). 46 


4.4 Logistical Aspects of Fellowship Experience 


4.4.1 Pre-Departure Orientation 

NSF organizes and pays for a pre-departure orientation in the Washington D.C. area, typically held in 
late March or early April. This orientation serves as an informational session as well as an 
opportunity for fellows to meet one another. Most of the fellows (87 percent) reported that they 


No exhibit. N=651, Missing=10 Awarded Applicants. Source: EAPSI Applicant Survey-Item El. 
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participated in this event. 47 Of the respondents who attended, more than half indicated that they 
received adequate information on living expenses (63 percent) and on the culture and rules of their 
host site (60 percent). 48 However, less than half of respondents were provided with adequate 
information on lodging (49 percent), healthcare expenses (45 percent), what not to do in the host 
site (44 percent), what to do in case of an emergency (41 percent), or the political situation of the 
host site (33 percent). To supplement the information provided at the orientation, fellows were 
encouraged to refer to other sources such as the EAPSI handbook, 49 EAPSI host site representatives, 
and the Internet to prepare for their visit. 

4.4.2 Administration of EAPSI in the Host Sites 

NSF administers the EAPSI program in the U.S. with co-sponsorship by at least one organization in 
each respective host site. Typically, each host site has a central program manager who is generally 
responsible for the overall day-to-day logistics and coordination of the program; only one location 
reported that host institutions are responsible for the administrative activities related to the 
program. 

Interviews with the program managers indicated that fellows typically participate in an additional 
orientation session offered by the foreign co-sponsoring organizations. Four out of the seven 
program managers interviewed specifically mentioned that they (or their staff) were responsible for 
running orientation sessions for the fellows upon their arrival in the host site. These sessions last 
between three days and one week and often include a "meet-and-greet" between the fellows and 
other researchers, embassy staff, and others. Orientation sessions also provide information and 
instruction to help fellows negotiate their time in the host country. According to the program 
managers interviewed, the orientation gives the fellows information about the research culture and 
environment and in some cases even includes language training. 

Fellows' living expenses at the host country are supported by the foreign co-sponsoring organization 
in the EAPSI region. Most program managers said that they administer the students' stipends and 
provide insurance. In one site, the program manager reported offering fellows some guidance on 
how to spend their stipend money (e.g., percentage that should be spent on rent versus language 
classes versus other miscellaneous expenses). Program managers from some sites reported a hands- 
on approach with the fellows, indicating that they help to arrange their flights, pick them up at the 
airport, and remain in contact with them during their stay. In some locations, program managers 
also facilitate communication between host researchers and the fellows. Program managers from 


No exhibit. N=887, Missing=51 Fellows. Source: EAPSI Applicant Survey-Item B6 

48 No exhibit. N=797, Missing=0 Fellows. This item was excluded for fellows whose response to B6=No (did 
not participate in the EAPSI pre-departure orientation. Responses do not sum to 100 because multiple 
responses were permitted. Source: EAPSI Applicant Survey-Item B6a 

49 There are site-specific handbooks that may serve as a resource for EAPSI fellows. The handbooks contain 
information on program guidelines and other topics such as: host institutions and researchers, travel and 
housing arrangements, safety and health, schedule of activities, and communication/reporting 
requirements. 
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other sites, however, appeared to provide support only if fellows run into problems during their 
stay. 

Program managers also described different types of support that the program provides the host 
researchers, which varied depending on the host site. In some cases, host researchers receive 
agency funds in exchange for hosting the fellows for the summer (one program manager noted that 
their site does not provide any financial support). Program managers from four sites said that host 
researchers are given an introduction to the program and advice about how to prepare for and host 
the fellows. In one location, for example, the program manager tells hosts in advance how to deal 
with potential problems (e.g., if the fellow needs to leave the country early), and recommends that 
the hosts encourage the students they regularly work with to engage in both research and cultural 
activities with the fellows. 

Program managers and agency officials reflected on their experiences with the EAPSI program, and 
highlighted some challenges that they faced with its administration. Some challenges revolved 
around funding and other limited resources. One agency official noted that securing the budget is a 
challenge. Another location would like to increase the size of the program, or at least obtain more 
funding for the current program, but lobbying efforts with the government have not been 
successful. Limitations of other resources also posed challenges. One program manager noted that, 
since they host students from other countries, there is competition for host institutions and living 
space. At least two other program managers agreed that there can be challenges associated with 
finding a residence. 

Program managers and agency officials cited few challenges from the perspective of the host 
researchers. One example of a challenge faced by host researchers was what to do if a student 
needed to leave early or postpone their stay. In one location, hosts did not know how to address 
this problem, so now the program manager's office informs the host institution about how to deal 
with this issue if it arises. Additionally, one agency official felt that it can be challenging for hosts to 
ensure that the fellows have a rewarding and fun research experience in such a short amount of 
time. 

One of the ways that program managers learn about program challenges and solicit 
recommendations for improvement is by administering an end-of-year questionnaire to fellows. The 
results of these questionnaires are often used to make program modifications at the local level. One 
of the program managers also reported that they learned how to help the host enhance the 
experience for the fellows, and that they can make improvements to the program based on host 
feedback. Another noted that one of his main responsibilities is to provide feedback from these 
surveys to host institutions, so they can make necessary changes. Finally, one program manager 
reported conducting a session to debrief on the last year's experiences, where fellows spoke about 
their research, and their experience with the program in general. However, the manager noted that 
they were unlikely to receive funding for these debriefs in future years. 
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4.4.3 Fellows’ Transition to the Host Site 

The costs of traveling to and from the host site are covered by the EAPSI award, but the fellows are 
responsible for making the travel arrangements themselves. Additionally, fellows may need to 
obtain a visa to cover the duration of their fellowship stay and may be responsible for securing their 
own housing accommodations. To facilitate a smooth transition, NSF recommends that fellows 
contact individuals in their host site such as the foreign program manager, their host, or their host 
institutional representatives for assistance and information on travel and housing. Based on 
responses from the host researchers, it appears that more than half were involved in helping their 
fellow obtain visas and make lodging arrangements (Exhibit 4.2). 

The hosts were asked to indicate the ease or difficulty of assisting their fellow with arranging 
lodging. More than half (59 percent) reported that it was easy and 27 percent experienced some 
level of difficulty. Similarly, over half of the hosts who helped the fellows to obtain a visa (55 
percent) found this easy and 5 percent experienced some level of difficulty. 

Furthermore, host researchers tended to take an active role in supporting fellows as they 
transitioned into the research activities at their designated site (Exhibit 4.2). A majority indicated 
that they assisted with ensuring that their fellow had access to buildings, labs, or other needed 
facilities (90 percent), arranging their fellow's office space (83 percent), and finding their fellow an 
appropriate research project (75 percent). More than three-quarters of the host respondents found 
that making these preliminary arrangements for the fellow was easy whereas 20 percent or less 
reported having some degree of difficulty with them. 
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Exhibit 4.2: Role of Hosts in Logistical Arrangements 


Arranging the fellow's access to 
buildings, labs, or other facilities 


Arranging the fellow's office space 


Finding the fellow an appropriate 
research project 

Providing supporting materials for the 
fellow's application 


Arranging the Fellow's lodging 


Providing assistance with obtaining a 
visa 


7.4 


15.5 


19.7 


110.4 15.5 


26.8 14.3 



9 40.6 


2.3 


1.5 


5.2 


0% 20% 40% 60% 80% 100% 

I Easy ■ Somewhat /Very difficult ■ N/A: Did not make this arrangement 


NOTES: N=656-660, Missing=5-9 Hosts. This item was only answered by hosts whose response to A2=l (indicated that they 
did host the EAPSI graduate student named in the survey). 

SOURCE: EAPSI Host Survey-Item B7 

4.4.4 Logistical Aspects of Fellowship 

Nearly all of the fellows (99 percent) agreed that they received notification of their EAPSI award 
with enough time to make the necessary travel arrangements to the host site. 50 Sixty percent or 
more of respondents were very satisfied with several logistical aspects of their EAPSI fellowship, 
including the amount of fellowship support (78 percent), duration of the fellowship (63 percent), 
research/laboratory facilities (63 percent), lodging (62 percent), and internet access (60 percent); 5 
percent or less expressed that they were very dissatisfied with any of these particular aspects of the 
fellowship (Exhibit 4.3). 


50 No exhibit. N=873, Missing=65 Awarded Applicants. Missing includes 'do not recall responses.' Source: 
EAPSI Applicant Survey-Item B3. 
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Exhibit 4.3: Fellows' Satisfaction with Logistical Aspects of Fellowship 


Fellowship support amount 


Fellowship duration 


Research / laboratory facilities at host institution 


Lodging 


Access to the internet 


78.0 


63.2 


61.8 


15.2 3.3 3.4 


25.8 7.5 3.4 


59.6 


29.2 3.7 4.3 


26.3 7.4 4.6 


25.1 10.2 5 


0% 20% 40% 60% 80% 100% 

Very satisfied Somewhat satisfied E Somewhat dissatisfied Very dissatisfied 


NOTES: N=644-645, Missing=16-17 Fellows. This item was only answered by fellows whose response to Dlc=l, 2, or 4 
(indicated that they were not still in graduate school as of October 1, 2010). 

SOURCE: EARS I Applicant Survey-Item E8 


Furthermore, only 11 percent of fellows 
reported that they encountered logistical 
difficulties during their fellowship stay 
and even fewer experienced specific 
difficulties with inadequate access to 
space, facilities, equipment, computers, 
resources and/or supplies (8 percent) or 
had legal or medical difficulties in the 
host site (1 percent). These challenges 
varied significantly by location, with the 
logistical challenges cited by a high of 48 
percent in one location to 6 percent in 
another, and inadequate access cited by a 
high of 40 percent in one location to a low 
of 4 percent in another. 51 


It is difficult for me to imagine a better foreign 
experience for a graduate student primarily 
because of the independence afforded by the 
EAPSi program. Unlike other foreign experiences 
for graduate students ([specific program] comes 
to mind) EAPSI is a blank slate for students to 
make the most of. This is critical I think for 
building independent self-reliant researchers. But 
also the contacts one can potentially make while 
on the EAPSI program can be resources that prove 
valuable for both sides long after the conclusion 
of an EAPSI experience. The support is very good 
also meaning that students can participate with 


\ little or no financial sacrifice. (EAPSI fellow) 

Like the fellows, their U.S. advisors were 
satisfied with the structure of the 

program; an overwhelming majority of advisors agreed that both the duration (8 to 10 weeks) and 
timing of the program (during the summer months) were appropriate (92 and 95 percent, 


No exhibit. N=648, Missing=13. This item was only answered by fellows whose response to Dlc=l, 2, or 4 
(indicated that they were not still in graduate school as of October 1, 2010). SOURCE: EAPSI Applicant 
Survey-Item E5. 
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respectively). 52 As discussed above, some foreign hosts, in contrast, did not see the duration of the 
fellowship as adequate: 29 percent of host respondents strongly agreed and 51percent agreed that 
the amount of time for the fellowship was too short for the proposed project. 53 


E 


.5 EAPSI Research Experience 


Fellows' research experiences, including their 
collaborations with foreign scientists at their 
host sites, are a central component of the EAPSI 
program. Fellows were asked to reflect on the 
research they conducted and their relationship 
with the host and other individuals they worked 
with during their fellowship. Likewise, hosts 
were asked about their experiences serving as 
an EAPSI host and working with the visiting 
fellow. Additionally, fellows' advisors and the 
host location representatives (program 
managers and agency officials) provided 
additional information and feedback on the 
program. Taken together, responses from the 
various respondent groups provide a 
comprehensive description of the research 
activities and collaborations that are associated with 
period (2000-2009). 


I wholeheartedly am supportive of the 
EAPSI program and greatly value its 
commitment to encouraging intellectual 
collaboration and cultural exchange 
across nations. Given the ever increasing 
globalization of science, fostering 
graduate student research experience at 
the international level is crucial for 
furthering both intellectual development 
and providing them with invaluable 
experience in conducting research in 
novel cultural contexts. (Advisor of EAPSI 
fellow) 


the EAPSI program during this study's time 


4.5.1 Research Environment at the Hosting Universities 

The hosts most commonly described their department or unit as one that encouraged international 
collaborations (82 percent) and invited foreign researchers to visit their institution for research- 
related purposes (68 percent) (Exhibit 4.4). Just over half of the respondents characterized their 
department or unit as encouraging faculty to work with graduate students (53 percent). 


No exhibit. N=628-629, Missing=16-17 Advisors. Source: EAPSI Advisor Survey-Items E5a, E5b. 

No exhibit. N=649, Missing=16 Hosts. This item was only answered by hosts whose response to A2=l 
(indicated that they did host the EAPSI graduate student named in the survey). Source: EAPSI Host Survey- 
Item C4. 
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Exhibit 4.4: Characteristics of the Department/Laboratory of Host Researchers 


Hosts 


(N=671) 

Encouraged international collaborations 

82.1% 

Hosted foreign researchers visiting their institution for research-related purposes 

67.7 

Encouraged faculty to work with graduate students 

53.2 

Promoted fellowships and other opportunities for researchers in their country to 

32.5 

conduct research in another country 

Provided financial support to faculty pursuing international collaborations 

21.3 

Rewarded faculty for developing international research partnerships 

20.1 

None of the above 

3.4 

Did not recall 

1.3 


NOTES: Missing=0 Hosts. 
SOURCE: Host Survey-Item A6 


4.5.2 Research Activities and Collaborations in the Host Site 

The survey examined whether the hosts were appropriately chosen. A majority of hosts agreed that 
that their research interests were well-matched with those of their fellow (82 percent), and that 
they and the fellow shared similar goals and expectations for the EAPSI experience (60 percent). 
However, less than half (48 percent) agreed that their work style was complementary to their 
fellow's. 54 

NSF and the foreign co-sponsoring organizations provide advice to the fellows on how to achieve a 
productive research experience. For example, the EAPSI handbooks encourage fellows to develop a 
research plan and have discussions with their host about its feasibility prior to their visit. 

The program emphasizes that fellows and hosts should be mutually committed to the research 
project, but there are no specific requirements as to how the research project should be supervised, 
or who should be involved in the research project. More than half (66 percent) indicated that their 
host directly supervised the work they did (Exhibit 4.5). At the same time, roughly a quarter of the 
fellows reported that others at the host institution provided direct supervision on their work, 
including graduate students (27 percent), staff scientists (22 percent), and junior faculty members or 
postdocs (21 percent). Less than 10 percent stated that their work was not supervised by anyone. 


No exhibit. N=662, Missing=3 Hosts. This item was only answered by hosts whose response to A2=l 
(indicated that they did host the EAPSI graduate student named in the survey). Source: EAPSI Host Survey, 
Item C6. 
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Exhibit 4.5: Individuals who Supervised Fellows' Work as Reported by Fellows 



Fellows 

(N=648) 

Host researcher 

65.6% 

Graduate student(s) 

26.7 

Staff scientist 

22.3 

Junior faculty member or postdoctoral fellow 

21.3 

Laboratory technician or other employee/worker 

6.8 

Was not supervised by anyone 

8.3 


NOTES: Missing=13. This item was only answered by fellows whose response to Dlc=l, 2, or 4 (indicated that they were 
not still in graduate school as of October 1, 2010). Responses do not sum to 100 because multiple responses were 
permitted. 

SOURCE: EAPSI Applicant Survey-Item E6 

Hosts also provided information about the individuals who directly worked on research with their 
fellow (Exhibit 4.6). The individuals most commonly cited by hosts as working with the fellows were 
hosts (80 percent), graduate students in the host's research group (66 percent), and other scientists 
in their research group (45 percent). Few hosts reported that fellows worked independently on their 
research without supervision (2 percent). 


Exhibit 4.6: Individuals Who Directly Worked on Research with Fellows as Reported by Hosts 


Fellows worked directly with... 

Hosts 

(N=664) 

1 [Host] worked directly with the fellow 

80.0% 

Graduate students in my research group 

65.5 

Research scientists in my research group 

44.7 

Postdoctoral fellows in my research group 

29.1 

Undergraduate students in my research group 

16.3 

Other visiting EAPSI fellows 

3.8 

Other individuals not listed above 

8.3 

No supervision but independently 

1.7 

None of the above 

0.3 


NOTES: Missing=l. This item was only answered by hosts whose response to A2=l (indicated that they did host the 
EAPSI graduate student named in the survey). Responses do not sum to 100 because multiple responses were 
permitted. 

SOURCE: EAPSI Host Survey-Item Cl 

Both fellows (Exhibit 4.7) and hosts (Exhibit 4.9) provided information about the activities on which 
they worked mostly independently and on which they collaborated. In general, a larger proportion 
of fellows reported that they worked on various research activities independently (between 26 and 
62 percent) as opposed to equally with the host or someone in the host's group (between 20 and 37 
percent). The activities fellows most commonly cited as being done equally with the hosts were 
interpreting results (37 percent), collecting data or carrying out simulations (33 percent), developing 
instrumentation (31 percent), and developing ideas or hypotheses (31 percent). 


Abt Associates Inc. 


4. Fellowship Activities 46 






Evaluation of the East Asia and Pacific Summer Institutes Program: Final Report 

Exhibit 4.7: Individuals Engaged in EAPSI Research Activities as Reported by Fellows 


Develop ideas, hypotheses, broad framework, 
or vision for research project 

Research relevant literature or research base 

Keep records, track supplies, resources 

Develop instrumentation, software equipment, 
or data collection processes 

Collect data or carry out simulations 
Analyze data or observations 
Interpret results 
Plan / develop followup work based on results 
Disseminate results (written / oral) 

■ Fellow and host equally ■ Mostly the fellow 


1 1 1 1 1 1 1 1 1 1 _ 1 
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41.7 


15.0 13.9 
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1 19.8 



52.0 



12.3 15.9 

1 1 1 1 1 1 1 1 


% 10% 20% 30% 40% 50% 60% 70% 80% 90%100% 


Mostly the host N/A - activity not part of research 


NOTES: N=624-633, Missing=28-37. This item was only answered by fellows whose response to Dlc=l,2, or 4 (indicated 
that they were not still in graduate school as of October 1, 2010). 

SOURCE: EAPSI Applicant Survey-Item E4 

Generally speaking, hosts' responses followed the pattern of responses from fellows (Exhibit 4.8). 
For example, on all but four research activities, a higher proportion of hosts reported that their 
fellow primarily worked independently (between 30 and 57 percent) as opposed to equally with 
them [hosts] (between 24 and 36 percent). The activities hosts most commonly cited as being done 
equally with the fellows were interpreting results (55 percent), developing ideas or hypotheses (47 
percent), planning or developing follow-up work based on results (42 percent), and analyzing data 
(36 percent). 
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Exhibit 4.8: Individuals Engaged in EAPSI Research Activities as Reported by Hosts 


Interpret results 

Develop ideas, hypotheses, broad framework, 
or vision for research project 

Plan / develop followup work based on results 
Analyze data or observations 
Research relevant literature or research base 

Keep records, track supplies, resources 

Develop instrumentation, software equipment, 
or data collection processes 

Collect data or carry out simulations 
Disseminate results (written / oral) 



8.4 4.61 0.9 


17.5 1.81 0.8 


14.9 11.9 I 1.1 


6.2 4.91 1.2 


12.4 2.11 1.4 


17.2 |1.9 


15.8 10.0 B2.9 


7.5 6.6 0.5 


5.8 13.7 ■ 3.1 


0% 20% 40% 60% 80% 100% 

I Fellow and host equally Mostly the fellow Mostly the host N/A - activity not part of research ■ Do not recall 


NOTES: N=646-653, Missing=12-19. This item was only answered by hosts whose response to A2=l (indicated that they did 
host the EAPSI graduate student named in the survey). This question matched one of the questions on the fellow survey 
except that the response category "do not recall" did not appear on the fellow survey. 

SOURCE: EAPSI Host Survey-Item C3 

Discrepancies between fellows' and hosts' perceptions may be explained by the tendency of 
individuals to overstate their own contribution to a joint task relative to other members of the 
group , 55 and biases in the way individuals themselves acting in accordance with an internal 
schematic encompassing their motives and goals . 56 Thus, it is possible that fellows perceived their 
role as active researchers, whereas hosts perceived their contributions as mentors, nurturing the 
fellow's research training through collaboration. 

4.5.3 Involvement of Fellows’ Home Institution 

During the EAPSI fellowships, many fellows maintained contact with their home institutions. About 
one-third of fellows reported that other researchers from their home institution became involved in 


Kruger, J., and Savitsky, K. 2009. On the genesis of inflated (and deflated) judgments of responsibility: 
Egocentrism revisited. Organizational Behavior and Human Decision Processes, 108 (1), 143-152; Ross, M. 
& Sicoly. F. (1979). Egocentric biases in availability and attribution. Journal of Personality and Social 
Psychology, 37, 322-336. 

Woike, B. 2008. A functional framework for the influence of implicit and explicit motives on 
autobiographical memory. Personality and Social Psychology Review, 12(2), 99-117; Conway, M.A. and 
Pleydell-Pearce, C.W. 2000. The construction of autobiographical memory in the self-memory system. 
Psychological Review, 107(2), 261-288. 
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the research project. These individuals included the fellow's graduate advisor (32 percent), other 
graduate students (7 percent), other faculty (4 percent), and undergraduate students at their 
graduate institution. 57 Similarly, the graduate advisors indicated that during the time their 
student/fellow was visiting the host institution, they provided ongoing academic support (70 
percent) and helped facilitate collaboration between the fellow/student and the host researcher (52 
percent - Exhibit 4.9). Advisors were less likely to provide the fellow/student with logistical, cultural 
or language support (16 percent) and to have visited him/her at the host institution (11 percent). 


Exhibit 4.9: Ongoing Support Provided by Graduate Advisor 



Advisors 

(N=644) 

Provided academic support to the fellow 

70.2% 

Helped facilitate collaboration between the fellow and the host scientist 

51.9 

Worked with the fellow to develop a publication based on research conducted at 
host institution 

37.9 

Provided logistical, cultural, or language support 

16.5 

Visited the fellow at the host institution 

11.3 

Other 

5.9 

None of the above 

18.2 


NOTES: Missing=0 Advisors. Responses do not sum to 100 because multiple responses were permitted. 
SOURCE: EAPSI Advisor Survey-Item Cl 


4.5.4 Fellows’ Satisfaction with Research-Related Activities at Host Site 


Most fellows were satisfied with several 
aspects of the EAPSI fellowship that pertained 
to their research and to interactions with their 
host (Exhibit 4.10), in particular the host's 
expertise in his/her field (81 percent very 
satisfied, 14 percent somewhat satisfied), the 
match between the fellow's research interests 
and the host (63 percent very satisfied, 26 
percent somewhat satisfied), and the level of 
the host's intellectual contribution to their 
joint research project (56 percent very 
satisfied, 28 percent somewhat satisfied). Less 
than 20 percent of fellows were dissatisfied 
with any specific area of the fellowship. 


I was very satisfied with the host 
institution and the researchers who 
worked there. People were very friendly 
and went out of their way to include me 
in excursions and meals, as well as in 
scientific discussions. The senior professor 
in the research group was very gracious 
and met with me several times to discuss 
research and science. He also asked me to 
review a manuscript for the journal for 
which he served as editor. (EAPSI fellow) 


No exhibit. N=648, Missing=13 Fellows. This item was only answered by fellows whose response to Dlc=l, 
2, or 4 (indicated that they were not still in graduate school as of October 1, 2010). Responses do not sum 
to 100 because multiple responses were permitted. Source: EAPSI Applicant Survey-Item E7 
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Exhibit 4.10: Fellows' Satisfaction with EAPSI Research-Related Activities at Host Site 


Host's expertise in his/her field 

Match between host's and my research interests 

Level of host's intellectual contribution to our 
joint research project 
Host's inclusion of me in research 
group/laboratory (meaningful collaboration) 

Host's efforts to help me meet other researchers 

Frequency of meetings with host 

Match between scope/ goals of project and 
duration of EAPSI 

Guidance or mentoring provided by host 
Quality of research I was able to conduct 


13.5 2.13.1 



80% 100% 


■ Very satisfied Somewhat satisfied ■ Somewhat dissatisfied Very dissatisfied 


NOTES: N=628-640, Missing=21-33. This item was only answered by fellows whose response to Dlc=l,2, or 4 (indicated 
that they were not still in graduate school as of October 1, 2010). 

SOURCE: EAPSI Applicant Survey-Item E8 

4.5.5 Host Feedback on Interactions with Fellow 


Similarly, a majority of hosts provided positive 
feedback on the EAPSI fellow they hosted. At 
least 90 percent of hosts strongly agreed or 
agreed that their EAPSI fellow demonstrated 
sufficient knowledge and expertise (45 percent 
strongly agreed, another 45 percent agreed) 
and integrated well with their research group 
(58 percent strongly agreed, 35 percent 
agreed). A similarly high proportion of hosts 
agreed that that their fellow was hard-working 
and dedicated (42 percent strongly agreed and 
46 percent agreed) and that their fellow spent 
sufficient time working on the project (38 
percent strongly agreed and 49 percent agreed, 


The fellow introduced a new 
experimental system and a new idea into 
my group. The experience working on a 
new experimental system was quite 
stimulating for me. She worked 
efficiently, quickly setting up 
experimental conditions to obtain 
reliable results, and made successful 
accomplishments in the short period of 
her stay. (EAPSI host) 


Exhibit 4.11). 
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Exhibit 4.11: Host Feedback on Interactions with Fellow 


Fellow integrated with staff/members of my 
research group 

Fellow had sufficient knowledge/expertise to be full 
participant in research collaboration with me 

Fellow was hard-working and dedicated 


Fellow spent sufficient time working on the project 

Fellow exercised appropriate caution in his/her 
approach to research 

Fellow was willing to take appropriate risks 
necessary for research 



■ Strongly agree Agree ■ Disagree Strongly disagree Does not apply or do not recall 


NOTES: N=645-651, Missing=14-20 Flosts. This item was only answered by hosts whose response to A2=l (indicated that 
they did host the EAPSI graduate student named in the survey). 

SOURCE: EAPSI Host Survey-Item C4 

Flosts rated EAPSI fellows favorably compared to other graduates students they worked with; 19 
percent of hosts reported that they were "much more satisfied," 26 percent were "somewhat more 
satisfied," and 37 percent were "equally satisfied" with their EAPSI fellow compared to other 
graduate students. 58 

In addition, many hosts described specific characteristics or skills of the fellow that they found 
particularly appealing. For example, hosts often referenced the fellow's motivation and initiative. 

She worked very hard indeed presenting great motivation to work in her research field. Though 
my team members spent lots of time and energy helping her to conduct field works, I know they 
learned much from her. (EAPSI host) 

[The fellow] was very motivated. He could work independently without the need of constant 
supervision. He worked hard. He was well trained and had sufficient knowledge in what he was 
going to do. He had sufficient cultural sensitivity and savvy to function in a different culture. He 
was a well-rounded person. It was a pleasant experience professionally and personally to host 
him. (EAPSI host) 


No exhibit. N=652, Missing=13 (Flosts). This item was only answered by hosts whose response to A2=l 
(indicated that they did host the EAPSI graduate student named in the survey). Source: EAPSI Applicant 
Survey-Item El 
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Other hosts commented on the fellow's interpersonal skills and thought they integrated into the lab 
well. 

[The fellow] brought a level of maturity to the research culture amongst the other graduate 
students in my lab at the time. He was supportive and collaborated on a number of other 
projects and fully integrated in to the laboratory. (EAPSI host) 

Hosts also were impressed with the academic and professional qualifications of the student. 

He brought in the skills that we did not have in our research group and applied it for the 
experiments in our facility. He was very hard working, and also helped out installation of some of 
our research equipment which falls into his expertise. (EAPSI host) 

4.5.6 Difficulties with EAPSI Research Activities and Collaboration 

Consistent with the high levels of satisfaction expressed by fellows, only a minority of fellows 
expressed difficulties related to their research activities and host interactions (Exhibit 4.12). 
Seventeen percent reported that they did not receive enough guidance from their host (or their 
host's research group) and less than 6 percent cited other difficulties related to their research 
activities, specifically being assigned a role less than merited by their skills or knowledge (5 percent), 
not receiving credit for their contributions to advancing a research project (1 percent), lack of 
respect given to their ideas (1 percent), and being assigned work that was someone else's 
responsibility (1 percent). Likewise, few hosts reported specific challenges collaborating on research 
with their EAPSI fellow (Exhibit 4.12). 


Exhibit 4.12: Difficulties with EAPSI Research Activities and Collaboration 


Fellows' Difficulties a 

Fellows 

Not enough guidance from host/host's research group 

16.6% 

My role on the project was less than that merited by my skills/knowledge 

5.4 

Not given credit for my contributions to advancing a project 

1.2 

1 felt that my ideas were not treated with respect 

1.2 

1 was asked to do work that was someone else's responsibility 

1.1 

Hosts' Difficulties b 

Hosts 

Fellow did not devote enough time/effort to collaboration 

7.9% 

Fellow had unanticipated gaps in his/her preparation to conduct research with me 

3.6 

Fellow worked too independently, did not work well as a collaborator or team 
member 

2.9 

Fellow needed too much guidance 

2.0 

Fellow lacked sufficient understanding of cultural norms in my country 

2.0 

The fellow and 1 had a difference of opinion about the direction of the research 

1.2 

Fellow was disrespectful, caused conflict with my research group 

0.8 

NOTES: 

a N=648, Missing=13 Fellows. This item was only answered by fellows whose response to Dlc=l, 2, 

or 4 (indicated that they 


were not still in graduate school as of October 1, 2010). Responses do not sum to 100 because multiple responses were 
permitted. 

b N=658, Missing=7 Hosts. This item was only answered by hosts whose response to A2=l (indicated that they did host the 
EAPSI graduate student named in the survey). Responses do not sum to 100 because multiple responses were permitted. 
SOURCE: EAPSI Applicant Survey-Item E5, EAPSI Host Survey-Item E3 
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In their interviews, host site representatives cited few challenges that arose from the research itself. 
Although one program manager stated that the main challenge faced by fellows is how to deal with 
the fact that, on occasion, their research does not go as planned. Another respondent noted that 
there are some instances where the host institutions are uncomfortable assigning a student to the 
requested researcher. This respondent suggested that it might be better to work with host 
institutions before putting out a list of researchers to participants, in order to minimize potential 
disappointments. 


.6 Professional and Cultural/Leisure Activities 


4.6.1 Fellows’ Engagement in Professional Activities 

In addition to engaging in research, fellows are encouraged to participate in a variety of other 
professional activities during their time at the host site. The most common activities reported by 
fellows (Exhibit 4.13) included visiting other educational or research institutions (72 percent), 
networking with colleagues from institutions outside their host institution (64 percent), and giving a 
presentation or talk to researchers (63 percent). 


Exhibit 4.13: Engagement in Professional Activities in the Host Site 


Fellows 

(N=651) 

Visit(s) to educational or research institutions other than my host institution 

71.5% 

Networking with colleagues from institutions other than my host institution 

63.5 

Gave a talk or presentation to researchers at my host site 

62.9 

Lectures, colloquia, seminars in my field 

54.3 

Visit(s) to businesses/industrial laboratories 

39.5 

Language courses or language study 

36.1 

Other professional activities not listed above 

5.5 

Did not attend or participate in any professional activities 

3.7 


NOTES: Missing=10 Fellows. This item was only answered by fellows whose response to Dlc=l, 2, or 4 (indicated that 
they were not still in graduate school as of October 1, 2010). Responses do not sum to 100 because multiple responses 
were permitted. 

SOURCE: EAPSI Applicant Survey-Item E3 

Most fellows were either very satisfied (52 percent) or somewhat satisfied (36 percent) with the 
professional connections they made during their fellowship; 4 percent were very dissatisfied and 8 
percent were somewhat dissatisfied. 59 


N=639, Missing=22 Fellows. This item was only answered by fellows whose response to Dlc=l, 2, or 4 
(indicated that they were not still in graduate school as of October 1, 2010). Responses do not sum to 100 
because multiple responses were permitted. Source: EAPSI Applicant Survey-Items E8 
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4.6.2 Fellows’ Engagement in Cultural/Leisure Activities 

One of the main purposes of the EAPSI 
program is to provide U.S. graduate students 
with an orientation to the society, culture, and 
language of their respective East Asian or 
Pacific host site. Nearly all fellows confirmed 
that they participated in cultural and leisure 
activities while abroad (Exhibit 4.14). Fellows 
took part in sightseeing (97 percent), 
exploring the landscape or geography (91 
percent), visiting museums (89 percent), and 
attending festivals, holiday events or religious 
ceremonies (70 percent) in their host site. 


Exhibit 4.14: Engagement in Cultural/Leisure Activities in the Host Site 


Fellows 

(N=651) 

Sightseeing 

96.5% 

Outdoor activities to explore the landscape/geography 

91.3 

Museums 

88.6 

Festivals, holidays, or religious ceremonies 

69.8 

Sporting events 

46.5 

Non-scientific lectures or presentations 

35.0 

Other cultural or leisure activities not listed above 

6.5 

Did not participate in any cultural activities 

0.1 


NOTES: Missing=10 Fellows. This item was only answered by fellows whose response to Dlc=l, 2, or 4 (indicated that 
they were not still in graduate school as of October 1, 2010). Responses do not sum to 100 because multiple responses 
were permitted. 

SOURCE: EAPSI Applicant Survey-Item E2 


The whole summer was really memorable; it is 
difficult to pinpoint precisely one activity/event 
that distinguishes itself from all the rest. 
However, I will say that staying with a host 
family for the weekend was a remarkable 
experience and is a component of the program 
I hope they maintain. (EAPSI fellow) 


4.6.3 Difficulties Related to the Culture and Language of Host Site 


Among all of the difficulties that fellows were 
asked about, communication or language issues 
were the most frequently cited (29 percent). As 
expected, these difficulties varied significantly 
by host site where a greater percentage of 
fellows who were in primarily non-English- 
speaking host sites reporting communication or 
language issues (ranging from 28 to 44 percent 
in these sites) compared to fellows who were in 
primarily English-speaking host sites ( ranging 
from 0 to 10 percent in these sites). 60 


Trying to be a good researcher when I was 
hindered by language difficulties. Though I 
did feel like i got a better grasp on the 
everyday language usage by having to use 
it— it didn't necessarily translate to 
professional level understanding of the 
language. (EAPSI fellow) 


60 No exhibit. N=648, Missing=13. This item was only answered by fellows whose response to Dlc=l, 2, or 4 
(indicated that they were not still in graduate school as of October 1, 2010). SOURCE: EAPSI Applicant 
Survey-Item E5. 
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Less than 3 percent of fellows cited any difficulties based 
percent), or cultural, or religious backgrounds (1 
percent). 61 Likewise, few hosts encountered cultural 
barriers to hosting their fellow. Merely 2 percent of 
host respondents cited that they had challenges with 
hosting their fellow because he/she lacked sufficient 
understanding of the cultural norms in their country. 62 
Rather, 33 percent of hosts strongly agreed and 55 
percent agreed that the scientific cultures of the U.S. 
and their respective location are similar, making 
productive collaboration possible. 


on gender (2 percent), racial or ethnic (1 

The most challenging aspect of my 
EAPSI experience was living in a 
cultural that was so unique from my 
own. There were cultural 
expectations and norms that had to 
be respected. This was not a 
problem, but was, at times, 
challenging. (EAPSI fellow) 


Only 8 percent of hosts reported experiencing challenges with their fellow's lack of familiarity with 
the language. When asked specifically if language was a barrier to the fellow's ability to interact with 
them [host] or others in their research group, 44 percent strongly disagreed and 45 percent 
disagreed that this was an issue (Exhibit 4.15). 


Exhibit 4.15: Language/Cultural Barriers Experienced by Hosts 


Scientific cultures of our countries are similar, 
making productive collaboration possible 


Language was a barrier to the Fellow's ability to 
interact with me and/or my group 


Differences in lifestyles and general cultures of our 
countries are a barrier to collaboration. 



0% 20% 40% 60% 80% 100% 

Strongly agree Agree Disagree Strongly disagree ■ Does not apply or do not recall 


NOTES: N=647-653, Missing=12-18 Hosts. This item was only answered by hosts whose response to A2=l (indicated that 
they did host the EAPSI graduate student named in the survey). 

SOURCE: EAPSI Host Survey-Item C4 


No exhibit. N=648, Missing=13 Awarded Applicants. This item was only answered by fellows whose 
response to Dlc=l, 2, or 4 (indicated that they were not still in graduate school as of October 1, 2010). 
SOURCE: EAPSI Applicant Survey-Item E5 

No exhibit. N=658, Missing=7 Flosts. This item was only answered by hosts whose response to A2=l 
(indicated that they did host the EAPSI graduate student named in the survey). SOURCE: EAPSI Host 
Survey-Item E3. 
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In their interviews, foreign counterpart representatives identified some cultural challenges faced by 
fellows and hosts. Two respondents identified a chief challenge faced by the fellows as how to deal 
with cultural differences. One program manager noted that while there are many similarities 
between [location] and the U.S., there are cultural differences which the students do not always 
take into account. This respondent finds it difficult that "the students in the U.S. don't really 
acknowledge that there are cultural differences between the U.S. and [location]. Economically, they 
are both developed countries. The cultural attitudes in the U.S. are the same in [location]; however, 
there are sometimes miscommunications." Another program manager noted that fellows can 
experience challenges due to "cultural differences in their daily lives," including differences in the 
research environment and living style between the U.S. and their host country. 


Recommendations 


4.7.1 Host Location Representatives 

Agency officials think that the NSF is meeting its goal of increasing collaboration with the countries 
participating in EAPSI. Respondents were generally satisfied with the level of communication and 
involvement of the NSF as well. Communication between NSF and the location officials usually 
occurs via email, although there were a couple of locations that reported s face-to-face contact with 
NSF staff, particularly when their orientations were held in the U.S. Some managers specifically 
reported that they appreciate NSF's flexibility in allowing each country to manage their own 
program. 

Respondents had a few suggestions for improvements regarding the role of NSF. The most popular 
suggestion, expressed by respondents from five locations, was for the NSF to establish a reciprocal 
program through NSF whereby their students could have opportunities to conduct research in the 
U.S. alongside American hosts. One respondent noted that she would like to increase the size and 
scope of the program in her country, but this was not possible due to funding constraints. 

A few other specific improvements or modifications were each mentioned by one or two individuals. 
One respondent mentioned that in the past NSF had sent a representative to visit the location, and 
continuing that practice would be welcome; the respondent recognized that the time required to 
travel such a large distance did introduce difficulties to this practice. Two locations would like to see 
more effort to increase awareness and recruit U.S. students to the program. 

Other recommendations were made regarding the fellow selection process. One respondent 
requested that NSF share information about the criteria used to select the EAPSI fellows; this 
respondent expressed interest in learning more about the selection process. Yet another 
respondent suggested a collaborative selection process that involved the foreign location office. 
Additionally, two respondents suggested changes to the timing of the selection process. Both 
indicated that it would be helpful for NSF to share materials with the sites earlier in the process; 
specifically, one representative expressed a desire to receive applicants' documentation in early 
February, as opposed to mid- to late February, in order to have enough time to complete the final 
participant list and contact hosts, while another representative noted that receiving the pre- 
orientation materials earlier in the year would be helpful. 


Abt Associates Inc. 


4. Fellowship Activities 56 



Evaluation of the East Asia and Pacific Summer Institutes Program: Final Report 


Only one suggestion was made regarding the structure of the fellowship. Specifically, the 
representative from one location suggested that NSF and the location agency should host a 
collaborative activity at the end of the program, during the 8th week, where the fellows could 
discuss their research and experiences with each other. 

Representatives from two locations expressed a desire for the NSF to provide additional post- 
fellowship support. One respondent suggested that it would be beneficial to get support from NSF 
on how to follow up with alumni in order to gather data on whether collaboration between 
scientists from the U.S. and the location increases in the long run. Another comment expressed 
interest in having NSF involved in campaigns for the alumni to return to the location, in order to 
further collaboration. Finally, one of the managers suggested that it might be useful to open the 
summer institute to undergraduate students, although this was not an official view of the agency. 

4.7.2 Survey Respondents 

Fellows, hosts, and advisors were asked to provide recommendations for the program. The most 
common suggestion made by fellows regarded the duration of the program. 

I went for only 2 months. It would hove been nice and more productive if I had gone for at least 
twice that long. My research really only got going after about 3 weeks on the ground.... It can 
take that long to understand the "lay of the land. " 

I would make it available for a longer period of time, especially for life sciences. Part of the issue 
with my project was that 8 weeks was not sufficient time to truly prepare and conduct a research 
project. 

Some suggestions related to logistics. For example, some fellows requested more assistance with 
finding housing, getting access to resources, distributing finances, taking language courses, etc. 

I would give the grant money to the students when they arrived in Japan. I would not depend on 
the host scientist to give the money to the students. It was uncomfortable asking for the money, 
especially in a culture like Japan. 

I think the logistics of sending students to [country] must be very difficult, but I felt very 
unsupported. For example, no one really knew how we were getting our visas. First it was as a 
group, then individually, etc. When I arrived at my host site, I was told I would need safety 
training before entering the lab, something that took almost two weeks to schedule and 
complete. 

I would better formalize the finding of housing. I was fortunate in that my host obtained visiting 
scientist housing at a very reasonable cost. Others in the program were spending excessive 
amounts on hotels because their hosts could not or did not find good housing. 

Some also suggested changes to the orientation programs or pre-departure activities. 

I'm not sure the orientation program in Washington DC is really necessary. It was fun but I didn't 
learn very much and it takes time while people are usually scrambling to get everything together 
for the summer away. You could consider transferring those resources to more orientation in the 
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host country. Another idea is to provide a small reimbursement for language study in the U.S. 
before the program starts. 

Encourage more communication between the host and the graduate student prior to starting the 
EAPSI program. Have students create a more explicit plan with their host so that there is a more 
clear deliverable at the end of the program. 
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5 Post-Fellowship Outcomes 


The primary goals of the EAPSI program are to (1) introduce students to East Asia and Pacific science 
and engineering in the context of a research setting, and (2) help students initiate scientific 
relationships that enable future collaborations with foreign scientists. 

This chapter examines the impacts of the program, by comparing EAPSI fellows' engagement in 
productive research collaborations with foreign collaborators and post-fellowship career 
characteristics to those of a matched comparison group of unfunded fellows. These career 
outcomes of EAPSI fellows are placed in a larger national context by comparing the EAPSI PhD 
fellows to data from the Survey of Doctoral Recipients and EAPSI master's fellows to respondents to 
the National Survey of Recent College Graduates with master's degrees. 

This chapter also explores the outcomes of the program from the perspectives of the foreign 
counterparts involved in administering the program, the EAPSI fellows, and the fellows' foreign 
hosts and U.S. advisors. In addition to outcomes for program participants, this chapter elaborates on 
the outcomes of the program beyond the direct participants. 

Specifically, this chapter answers the following questions: 

• Does the extent to which former fellows engage in international collaborations differ from that 
of unfunded applicants? 

• Do fellows' post-award career activities and job characteristics differ from those of unfunded 
applicants and other STEM graduates in the U.S.? 

• What do program participants view as the outcomes of the program? 

• Do the outcomes of program participation extend beyond the direct participants? 


5.1 Key Findings 


• Foreign locations become involved in EAPSI to increase research collaborations with the U.S., to 
establish a research network and ongoing relationships among scientists, and to provide 
younger researchers from the U.S. with an opportunity to understand the R&D activities in their 
countries. 

• All EAPSI fellows would recommend the fellowship to another graduate student seeking an 
international experience, 92 percent of advisors would or have recommended the program to 
other graduate students, and 79 percent of hosts would or have recommended hosting an EAPSI 
fellow to others. 

• Unfunded applicants continue to pursue other avenues for their international experiences. 

EAPSI fellows and unfunded applicants with PhDs held a similar number of international 
postdoctoral fellowships (0.32 versus 0.28 fellowships, respectively), yet fellows were less likely 
than unfunded applicants to have been employed outside the U.S. (13 versus 39 percent, 
respectively). 
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• Forty percent of fellows and 35 percent of unfunded applicants report working with individuals 
located in other countries. Among those who work with a collaborator in a foreign country, 
unfunded applicants were more likely to claim that their collaboration included joint 
publications or jointly developed products (82 versus 92 percent). 

• Compared to unfunded applicants, fellows reported a higher number of publications co- 
authored with a foreign collaborator (2.2 versus 1.6 publications) and a larger proportion of 
publications that include a foreign co-author (22 versus 17 percent), suggesting that fellows 
have more productive collaborations with foreign counterparts than do unfunded applicants. 

• Fellows were also more likely than unfunded applicants to engage in activities to foster 
international collaborations (40 versus 30 percent, respectively). 

• Overall, fellows and unfunded applicants were similar along a few of the education and career 
outcomes examined in this study; both groups were equally likely to have obtained a master's 
degree (37 versus 38 percent) or PhD as their highest degree (53 versus 48 percent) and had a 
similar number of total publications over a period of time following program participation (9.8 
versus 9.5 publications). 

• Benchmarking analyses comparing career outcomes of EAPSI fellows to national estimates for 
science and engineering PhD graduates found that EAPSI PhD fellows were more likely to be 
working with individuals in other countries (80 percent of fellows holding a PhD versus 31 
percent of the SDR sample). 

• A majority of fellows perceived their participation in EAPSI as having expanded their education 
and career opportunities. Among fellows no longer in graduate school, 78 percent felt it 
qualified them for a broader range of opportunities while a minority of fellows felt that their 
EAPSI participation had resulted in a lost opportunity; less than 1 percent felt it had constrained 
their opportunities, and 11 percent felt it had delayed their degree completion. 

• Just over half of fellows who were no longer in graduate school indicated that their participation 
led to valuable connections to researchers outside the U.S. (52 percent) and made them more 
competitive for jobs (51 percent). 

• Advisors also found the program successful in helping fellows further their research and 
collaborations abroad by helping them gain exposure to research in another country (86 
percent), advancing their research agenda (78 percent), and establish collaborations with 
researchers outside the U.S. (75 percent) 

• Foreign agency representatives identified benefits of the program for fellows, other students in 
the host lab or institution, host researchers, and the host institutions and countries. They 
commonly cited increased opportunities for collaboration and networking as benefits of 
participation. 

• A majority of the fellows indicated that their relationship with the host has continued beyond 
the fellowship period, either through subsequent research collaborations with the host (20 
percent), and/or additional communications (60 percent). Flosts had similar responses to the 
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fellows, as 29 percent stated that they had further collaborated with the fellow and 48 percent 
had communications with the fellow post-fellowship. 

• More than half of the fellows engaged in activities where they extended the benefits of their 
EAPSI participation with others, particularly by sharing resources they collected or tools 
developed during their fellowship (59 percent) and teaching others about the research methods 
they learned during that time (56 percent). 


5.2 Overall Perceptions of EAPSI 


5.2.1 Host Site Representatives’ Descriptions of Progress toward Program Goals 

The agency officials who were interviewed described the program goals from their perspectives and 
the progress the program made toward these goals. They reported that their countries originally 
became involved in EAPSI to increase research collaborations with the U.S., to establish a research 
network and ongoing relationships between scientists, and to provide younger researchers from the 
U.S. with an opportunity to understand the R&D activities in their countries. Their reasons for 
participating align with NSF goals for the EAPSI program. 

Each program manager also noted that the primary goal of their EAPSI program was to increase 
collaboration between the U.S. and their country. As one program manager explained, "the goal is 
for us to provide the opportunity for faculty and younger generations to engage in scientific 
dialogue, face-to-face discussion, personal exchange of ideas or issues in science and technology, 
research, and career development. We want this to be a life-long experience." In this way, the 
program is mutually beneficial to both the U.S. and the foreign location, the perspectives of 
emerging researchers are broadened, and lasting professional relationships are established. 

Almost all the program managers felt that this goal was being met successfully. One respondent felt 
that it was still too early to accurately tell whether EAPSI has increased collaborations between U.S. 
researchers and researchers in the location. This respondent also raised a question about whether 
the dosage of the EAPSI experience was sufficient, indicating that the research experience needs to 
be longer in order to truly facilitate a collaborative working relationship, stating, "for participants to 
have a true understanding of R&D overseas, it is necessary for them to spend an extended stint, of 
perhaps one year in [location]. Otherwise, it could just be a sponsored holiday to the far east." 
Another respondent noted that it is difficult to attribute an increase in collaboration to EAPSI in 
particular, since the foreign agency engages in many activities to increase international 
collaboration. 

A strategy cited by one location for maintaining collaboration is to establish an alumni network for 
the fellows. This "allows them to maintain a collaborative relationship with the [location] even after 
they are recruited elsewhere." Another respondent indicated that while the goal of increased 
collaboration was being met, they would like to build a new mechanism for researchers in the 
location and the U.S. to communicate and exchange information after their visit. 
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5.2.2 Respondents’ Satisfaction with the Program 

Survey respondents were also supportive of the program. All of the EAPSI fellows said that they 
would recommend the fellowship to another graduate student seeking an international experience. 
Moreover, nearly all fellows would also recommend their host site (99 percent) and their host 
scientist (91 percent). Similarly, most advisors (92 percent) would recommend or have 
recommended the program to other graduate students. Satisfaction among the hosts was 
somewhat lower, with 79 percent of respondents saying that they would recommend (or have 
already recommended) hosting an EAPSI fellow to others. 

Open-ended comments provided by respondents offer more insights on the benefits of the 
program. When asked to elaborate why they would or would not recommend the program, some 
fellows emphasized the importance of learning how research was performed in foreign countries. 

I consider EAPSI fellowship a great opportunity for professional collaboration, life-long 
professional connections, a way to grow professionally. Connecting to another culture and 
understanding its roots help us to become better human beings. The program broadens one's 
perspectives and views of the world. It creates a different (from home country) view of potential 
world problems and the ways to solve them. Overall, it is a great experience for any graduate 
student to visit another country not as a pure tourist, but as a researcher, to dive into a project 
and work side by side with the foreign colleagues. (EAPSI fellow) 

Others also mentioned the contribution the program made to their growth as individuals and the 
benefits of being exposed to other cultures. 

I was much more narrow minded before my experience. I guess I was afraid of the world outside 
my little bubble. Being forced to get out there and see if I could make it, with Uncle Sam 
propping me up, was the best growth experience of my life. Often I need to reach back to those 
lessons and remember what I learned. (EAPSI fellow) 

It was an incredible experience! It provided an opportunity to work and learn in a different 
country as well as providing orientation to country and culture. The crash course in language, the 
field trip and other activities like the homestay experience were awesome! (EAPSI fellow) 

Fellow advisors would recommend the program because it provides students with valuable and 
broader experiences and perspectives. 

I think experience in other labs, and international experience in particular, can be very valuable 
to the development of graduate students as scientists. (Advisor to EASPI fellow) 

I have personally seen many benefits to our students. It provides a new perspective and 
appreciation of the research culture in different countries and opens much needed 
collaborations. (Advisor to EASPI fellow) 

I strongly believe that international experiences are a very valuable component of a student's 
education. International exchange opens the mind and promotes diversity and creativity. 

(Advisor to EASPI fellow) 
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US. students often lack international experience, and as a result can become provincial or 
intellectually limited. EAPSI offers a very good controlled but useful research-abroad experience. 
(Advisor to EASPI fellow) 

Advisors also described the value of forging collaborations with foreign colleagues and institutions. 

I think the EAPSI program is one of the best programs at NSF for fostering a global, diverse and 
international scientific workforce in the US. I cannot emphasize how much this program has 
been valuable for building positive international research collaborations as well as further 
scientific discovery, particularly in my research area [disciplines], (Advisor to EASPI fellow) 

I believe there is enormous value in international collaborations, and initiating these as early as 
possible. Students have the opportunity to favorably compare themselves with students 
educated in other systems/cultures. Students broaden their perspective, are introduced to ideas 
and approaches outside the "home." Students establish life-long collaborations/associations, 
often with scientists much senior to themselves, opening future postdoctoral/career 
opportunities. (Advisor to EASPI fellow) 

Also, advisors noted that the program improved fellow career options. 

Working at a foreign institution, with different research supervisors, is an excellent experience, 
that broadens both their intellectual perspective, and also gives them different research 
experiences that helps further their careers. (Advisor to EASPI fellow) 

The benefits of the program from the hosts' perspective included scientific, cultural, and linguistic 
benefits to their students. 

It is very important to have our students and staff be exposed to students from other prominent 
research groups in our area to appreciate the strengths and weaknesses in our own approaches. 

[The fellow's] coming to my lab has offered an opportunity for my group to speak English and 
provided a window to know how a U.S. graduate student's attitude toward doing science. 


5.3 Impact of the Program on Fellows 


The impact of the EAPSI program was examined by comparing outcomes for EAPSI fellows to a 
rigorously matched group of unfunded applicants (using propensity score matching methods 
detailed in Appendix C) to identify the unique effects of the program. 

Two sets of outcomes were tested: first, a set of outcomes relating to fellows' subsequent 
engagement in international research and collaboration, and a second broader set related to 
careers . 63 For each outcome, the impact of EAPSI was estimated for a matched group of awardees 
and non-awardees, controlling for number of years since EAPSI application, under-represented 
minority status, gender, and where applicable, the number of pre-award publications and field of 


EAPSI awarded applicants were included only if they had completed their EAPSI fellowship by October 1, 
2010 . 
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study. Exhibits display the adjusted (estimated) means for awardees and non-awardees, the 
estimated impact, the standard error, and the p-value. For specifics on model specifications and 
standard error calculations, see Appendix C. 

5.3.1 International collaborations 

Exhibit 5.1 below shows the impact of the program on international collaborations, comparing the 
extent to which former fellows' engagement in international collaborations differs from that of 
unfunded applicants. Differences were investigated in the number of international postdoctoral 
fellowships, employment outside the U.S., engagement in work with individuals located in other 
countries, characteristics of work with individuals in other countries, publications with foreign 
collaborators, mentoring of individuals conducting research abroad, and activities to foster 
international collaborations. 

Graduate students apply to EAPSI because they are interested in pursuing an international research 
experience. Several findings suggest that unfunded applicants continue to pursue other avenues for 
their international experiences. Among EAPSI applicants who held a PhD, EAPSI fellows and 
unfunded applicants held a similar number of international postdoctoral fellowships 64 (0.32 
fellowships for EAPSI fellows versus 0.28 fellowships for the unfunded applicants; the difference was 
not statistically significant). Further, EAPSI fellows were less likely than unfunded EAPSI applicants to 
have been employed outside the U.S. (13 percent versus 39 percent, respectively), a statistically 
significant difference of 26 percentage points. 

More fellows than unfunded applicants reported that they currently work with individuals located in 
other countries (40 versus 35 percent), although this difference was not statistically significant. 
Among those who work with a collaborator in a foreign country, fellows compared to unfunded 
applicants were less likely to report that their collaboration included joint publications or jointly 
developed products (82 percent versus 92 percent), a statistically significant difference. 

Despite how individuals characterized their collaborations, EAPSI fellows' collaborations with foreign 
collaborators resulted in greater productivity as measured by publications. Fellows reported a 
higher number of publications co-authored with a foreign collaborator (on average, 2.2 publications 
for fellows and 1.6 publications for unfunded applicants), a statistically significant difference of .6 
publications. Further, a larger proportion of the publications of fellows, than of unfunded applicants, 
include a foreign co-author (22 versus 17 percent), a statistically significant difference of 5 
percentage points. 

There was also evidence that the influence of the program extended beyond the immediate 
participants, as fellows were also more likely than unfunded applicants to engage in a series of 
activities to foster international collaborations among others (40 percent versus 30 percent, 
respectively), a 10 percentage point difference that was statistically significant. 


64 This included only postdoctoral fellowships abroad, thus international postdoctoral fellowships is a 
measure of internationalization, not of employment. 
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Exhibit 5.1: Comparison of Fellows and Unfunded Applicants on Career International Activities 




Unfunded 





Awardee 

Applicants 


Impact 



Adjusted 

Adjusted 

Impact 

Standard 


Outcome 

Mean 

Mean 

Estimate 

Error 

p-value 

Work with individuals outside the U.S. 


Total number of international postdoctoral 
fellowships 3 

0.32 

0.28 

0.04 

0.05 

0.469 

Employment outside the U.S. since [year marking 
end of fellowship period] b 

0.13 

0.39 

-0.26** 

0.04 

<0.000 

In current job, works with individuals located in 
other countries c 

0.40 

0.35 

0.05 

0.04 

0.218 

Type of current work with individuals in other 
countries includes joint publications and/or 

0.82 

0.92 

-0.09* 

0.05 

0.040 


jointly-developed products e 


Publications with foreign collaborator 

Number of publications co-authored with a 
foreign collaborator d 

2.19 

1.60 

0.59* 

0.25 

0.016 

Proportion of publications co-authored with a 
foreign collaborator d 

21.94 

16.65 

5.29** 

1.95 

0.007 

Fostering international collaboration 

Has mentored others from the U.S. traveling to 
another country to conduct research b 

0.24 

0.25 

-0.01 

0.04 

0.764 

Conducted activities to foster international 
collaboration 3 

0.40 

0.30 

0.10** 

0.04 

0.010 


*p<.05, **p<.01, ***p<.001 

EXHIBIT READS: EAPSI awarded applicants (who had completed their EAPSI fellowship by October 1, 2010) held more 
international postdoctoral fellowships then unfunded EAPSI applicants: 0.32 fellowships for EAPSI fellows versus 0.28 
fellowships for the unfunded applicants (a difference of 0.04 fellowships); this difference was not statistically 
significant. 

13 percent of EAPSI fellows had been employed outside the U.S. compared with 39 percent of unfunded applicants; this 
difference was statistically significant. 

NOTES: 

3 Because this question was limited only to those who had completed a PhD by October 1, 2010 we did not use highest 
degree achieved by October 1, 2010 as a covariate in our model. N=471 (394 Awarded, 74 Unfunded Applicants); 
Missing=3. Answered by applicants whose highest degree completed as of October 1, 2010 was a doctoral degree. 
b N=644 (528 Awarded, 116 Unfunded Applicants); Missing=13. Answered by applicants who were not enrolled in 
graduate school during the week of October 1, 2010. 

c N=623 (507 Awarded, 116 Unfunded Applicants). Answered by applicants who were employed during the week of 
October 1, 2010 

d N=875 (706 Funded, 169 Unfunded); Missing=97. 

6 N=633 to 635 (517 to 520 Awarded, 113 to 118 Unfunded Applicants); Missing=22 to 24. Answered by applicants who 
were not enrolled in graduate school during the week of October 1, 2010. 

SOURCE: EAPSI Applicant Survey-Items Dlb, D4, D4a, D4b, D6, D8, D10 

5.3.2 Educational and Career Outcomes 

While the program is not designed to directly affect education and career outcomes, an unintended 
consequence of participation might be delays in degree attainment or career progression resulting 
from the time spent conducting research abroad. The study did not find that educational 
progression was delayed, as fellows and unfunded applicants were equally likely to have obtained a 
master's degree (37 versus 38 percent) or a PhD as their highest degree (53 versus 48 percent) (no 
exhibit). Further, the research productivity of fellows and unfunded applicants, as measured by the 
number of their post-application publications, was similar (9.8 versus 9.5 publications, Exhibit 5.2). 
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Other career outcomes were specific to only small subsets of respondents, so findings should not be 
generalized to the larger group of respondents (Exhibit 5.2). Specifically: 

• Among those who reported having received an award as a principal or co-principal 
investigator, EAPSI fellows were significantly less likely than unfunded EAPSI applicants to 
report that their self-identified most prestigious research grant, award, or honor was from 
an international organization or foreign government: 12 percent of EAPSI fellows versus 30 
percent of the non-awardees. 

• Among applicants who were employed at an institution of higher education, fellows were 
more likely to be in a position that involved research, 85 percent of fellows versus 71 
percent of unfunded applicants, a statistically significant difference of 14 percentage points. 

• Among those individuals who were employed at institutions with a faculty rank system, 
unfunded applicants were significantly more likely to be in a tenure track position (56 versus 
40 percent). 

• Among those individuals in institutions with a tenure track system, tenure rates were similar 
between the two groups (4 percent versus 3 percent). 
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Exhibit 5.2: Comparison of Fellows and Unfunded applicants on Career Outcomes 




Unfunded 





Awardee 

Applicants 


Impact 



Adjusted 

Adjusted 

Impact 

Standard 


Outcome 

Mean 

Mean 

Estimate 

Error 

p- value 

All applicants 

Number of post-application publications a 

9.83 

9.51 

0.32 

0.85 

0.709 

Subset of applicants 

Most prestigious grant, award, or honor for research 
was from an international organization or foreign 
government b 

0.12 

0.30 

-0.18** 

0.05 

0.001 

Currently holds a research faculty, scientist, associate 
or fellow position at a 4-year college/university, 
medical school, or university-affiliated research 
institute c 

0.85 

0.71 

0.14* 

0.06 

0.010 

Currently has faculty rank of Assistant, Associate or 
Full Professor d 

0.40 

0.56 

-0.16* 

0.07 

0.021 

Currently has tenure f 

0.04 

0.03 

0.01 

0.05 

0.829 

*p<.05, **p<.01. ***p<.001 


EXHIBIT READS: EAPSI awarded applicants (who had completed their EAPSI fellowship by October 1, 2010) on average 
had 9.8 publications since completing EAPSI versus 9.5 publications for the unfunded EAPSI applicants (a difference of 
less than 1 publication). This difference was not statistically significant. Note, the count of publication included peer- 
reviewed journal articles, peer-reviewed conference publications (e.g. abstracts, conference papers, posters, and book 
chapter(s)). 

Among those individuals who reported receiving a research grant, award or honor, twelve percent of EAPSI awarded 
applicants (who had completed their EAPSI fellowship by October 1, 2010) reported that their most award was from an 
international organization or foreign government, as opposed to 30 percent of unfunded applicants. This difference was 
statistically significant. 

NOTES: 

a N=879 (709 Awarded, 170 Unfunded Applicants), Missing=93. 

b N=291 (243 Awarded, 48 Unfunded Applicants); Missing=12. Includes applicants who had received grant(s) (as a 
principal investigator or co-principal investigator), prestigious awards or honors based on their research.) 
c N=189 (151 Awarded, 38 Unfunded Applicants). Includes applicants who were employed at an educational institution 
during the week of October 1, 2010. 

d N=148 (127 Awarded, 21 Unfunded Applicants). Includes applicants who were employed at an educational institution 
that has a faculty rank system. 

6 N=125 (104 Awarded, 21 Unfunded Applicants). Includes applicants who were employed at an educational institution 
with a tenure system. 

SOURCE: EAPSI Applicant Survey-Items D2a, D2c, D2d, D5b, D6. 
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There were some differences in the types of employers of EAPSI fellows versus unfunded applicants. 
EAPSI fellows were less likely to be employed in an educational institution (50 percent of fellows 
versus 62 percent of unfunded applicants) and in state or local government (1 percent versus 6 
percent); these differences were statistically significant. 


Exhibit 5.3: Comparison of Employers of Fellows and Unfunded Applicants 


Employed in 

Awardee 

Adjusted 

Mean 

Unfunded 

Applicants 

Adjusted 

Mean 

Impact 

Estimate 

Impact 

Standard 

Error 

P-value 

Educational Institution 

0.50 

0.62 

-0.11* 

0.05 

0.024 

Self Employed 

0.04 

0.01 

0.03 

0.02 

0.097 

Private Sector 

0.31 

0.22 

0.08 

0.05 

0.062 

State or Local Government 

0.01 

0.06 

-0.04* 

0.02 

0.046 

Federal Government 

0.12 

0.09 

0.03 

0.03 

0.378 

Other 

0.01 

-0.00 

0.02 

0.01 

0.264 


*p<.05, **p<.01, ***p<.001 

EXHIBIT READS: 50 percent of EAPSI awarded applicants (who had completed their EAPSI fellowship by October 1, 2010) 
were employed at an educational institution, as opposed to 62 percent of unfunded applicants. This difference was 
statistically significant. 

NOTES: N=599 (456 Awarded, 143 Unfunded Applicants), Missing=0. These items were answered by applicants who, 
during the week of Oct 1 2010, were working for pay or profit in a non-postdoctoral appointment. N 
SOURCE: EAPSI Applicant Survey-Items D2 and D3. 


Benchmark to National Data 


To understand the EAPSI fellows' outcomes within a broader context, the outcomes of EAPSI fellows 
were compared to national estimates for science and engineering graduates. Specifically, the 
outcomes of EAPSI PhD graduates were compared against national estimates from the Survey of 
Doctoral Recipients (SDR), and the outcomes of EAPSI master's graduates were compared against 
estimates of individuals who had completed master's degrees in the National Survey or Recent 
College Graduates (NSRCG). 65 

These comparisons are descriptive in nature and are not intended to address the impacts of the 
EAPSI program. Thus, findings should be interpreted with an understanding that there may be 
uncontrolled initial differences between EAPSI fellows and those who responded to the SDR and the 
NSRCG. Note that the EAPSI benchmarking estimates below are different from the EAPSI impact 
estimates above because the master's students and doctoral students are combined in the impact 
analyses above. Also, in the benchmarking analyses covariates are grand mean centered using the 
SDR and EAPSI samples (or the NSRCG and the EAPSI samples) versus impact analysis where the 
covariates are grand mean centered using the EAPSI sample only. 


Similar comparisons that compared both fellows and unfunded EAPSI applicants to national estimates are 
included in Appendix F. 
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5.4.1 Benchmark Comparison of EAPSI Fellows to National Samples 

EAPSI fellows were compared to the national samples on the frequency with which they worked 
with individuals in other countries (Exhibit 5.4 and Exhibit 5.5). EAPSI PhD fellows were more likely 
to be working "for pay or profit" (almost 100 percent of EAPSI PhD awardees versus 94 percent of 
the SDR sample, a statistically significant difference). EAPSI fellows with PhDs were significantly 
more likely to be working with individuals in other countries, compared to their counterparts 
nationally (80 percent of EAPSI PhD fellows versus 31 percent of the SDR sample; a statistically 
significant difference). 


Exhibit 5.4: Characteristics of Work of EAPSI Fellows and SDR Sample during Reference Week 



*p<0.05, **p<0.01, ***p<0.001 

NOTES: Currently employed: This item was answered by former EAPSI fellows who had completed their EAPSI fellowship 
(including any U.S.-based "re-entry period") as of October 1, 2010 (N=503, Missing=0) and by SDR 2008 respondents who 
had completed a PhD by October 1, 2008 (N=29,974, Missing=0). In current job, works with individuals in other countries: 
This item was answered by former EAPSI fellows who had completed their EAPSI fellowship (including any U.S. -based "re- 
entry period") as of October 1, 2010 and were employed during the week of October 1, 2010 (N=497,Missing=0) and by 
SDR 2006 respondents who had completed a PhD by April 1, 2006 and were employed during the week of April 1, 2006 
(N=27,119, Missing=0). This item was not included in the SDR 2008 wave. 

SOURCE: EAPSI Applicant Survey-Items Dla, Die, D2 and D4. SDR Survey 2006-Items A1 and A27 


Fellows whose highest degree at the time of the survey was a master's degree were equally likely to 
be working with individuals in other countries compared to master's graduates in the NSRCG sample 
(25 percent of EAPSI master's fellows versus 20 percent of the master's NSRCG sample; not a 
statistically significant difference. Exhibit 5.5). 
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Exhibit 5.5: Characteristics of Work of EAPSI Master's Fellows and NSRCG Sample 


Worked with individuals in other country 


24.8 

20.0 


■ EAPSI 

■ NSRCG 


0.0 20.0 40.0 60.0 80.0 100.0 


NOTES: Reference week was October 1, 2010 for EAPSI fellows and April 1, 2006 for the NSRCG sample. The NSRCG sample 
excluded individuals living outside of the United States. Difference is not statistically significant at the p<.05 level. 

EAPSI N=149 Missing=0. This item was answered by fellows who as of October 1, 2010 had completed EAPSI; NSRCG 
N=3,973 Missing=0. This item was answered by NSRCG respondents who had completed a master's degree by April 1, 

2006. 

SOURCE: EAPSI Applicant Survey-Items Dla, Die, and D4. NSRCG Survey 2006-Items B1 and B27. 


Additional comparisons were conducted to investigate the differences in the work settings between 
EAPSI PhD fellows and national estimates for similar S&E graduates (Exhibits 5.6). EAPSI PhD fellows 
were more likely to be employed in an educational institution (75 versus 45 percent) and less likely 
in state or local government (1 versus 3 percent). EAPSI PhD fellows working at educational 
institutions were less likely, however, to be on a tenure track (31 versus 50 percent, data not 
shown). 


Exhibit 5.6: Employers of EAPSI PhD Fellows and SDR Sample during Reference Week 



*p<0.05, * **p<0.01, ***p<0.001 

NOTES: This item was answered by EAPSI fellows who had completed their EAPSI fellowship, had earned a PhD, and were 
employed as of October 1, 2010 (N=378, Missing=0) and by SDR 2008 respondents who had completed a PhD and were 
employed during the week of October 1, 2008 (N=26,191, Missing=0). Items from which these data derive differed slightly 
between the EAPSI Applicant Survey and the SDR 2008; thus, Local Government (city, county, school district) and State 
Government (including state colleges/universities) were combined into a single category for both groups; and U.S. Federal 
Government and U.S. Military service, activity duty or Commissioned Corps (e.g., USPHS, NOAA) were combined. 

SOURCE: EAPSI Applicant Survey-Items Die, D2 and D3. SDR Survey 2008-Items Al, All and A12. 
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Among those working in institutions of higher education, EAPSI fellows were more likely to have 
research faculty/scientist/associate/fellow positions (82 percent of fellows versus 43 percent from 
the SDR sample, a significant difference) (Exhibit 5.7). 

Exhibit 5.7: EAPSI PhD Fellows and SDR Samples' Positions in Academic Institutions 


President, Provost, or Chancellor 

Dean (any level), department head or chair 

Research faculty, scientist, associate or 
fellow 

Teaching faculty 
Adjunct faculty 
Other 


0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 

*p<0.05, **p<0.01, ***p<0.001 

NOTES: The academic positions are not mutually exclusive: Individuals could select more than one response. For example, 
individuals could hold a research faculty position and a teaching faculty position simultaneously. 

These items were answered by EAPSI fellows who had completed their EAPSI fellowship and had earned a PhD, were 
working at an educational institution (other than in a postdoctoral position) during the week of October 1, 2010 and who 
did not report working in a preschool, elementary, middle, or secondary school or system (EAPSI N=160, Missing=0) and by 
SDR respondents who had completed a PhD, who were working in an educational institution during the week of October 1, 
2008, and who did not report working in a preschool, elementary, or secondary school system (SDR N=ll,773, Missing=0). 
SOURCE: EAPSI Applicant Survey-Items Die, D2, D2a and D2c. SDR Survey 2008-Items Al, A12, and A14. 

Similar comparisons of employers were performed for EAPSI fellows with masters' degrees. Fellows 
were significantly less likely to be self-employed or business owners (1 versus 5 percent) and less 
likely to be in the private sector (56 versus 67 percent) (Exhibit 5.8). 



I EAPSI PhD 
SDR 
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Exhibit 5.8: Employers of EAPSI Master's Fellows and NSRCG Sample during Reference Week 


Educational institution 


Self-employed/business owner 
Private sector 


Local/state government 
Federal government 



■ EAPSI 

■ NSRCG 


*p<0.05, **p<0.01, ***p<0.001 

NOTES: Reference week was October 1, 2010 for EAPSI fellows and October 1, 2008 for the NSRCG sample. This item was 
answered by EAPSI awardees who had completed their EAPSI fellowship, whose highest degree earned was a master's 
degree and were employed as of October 1, 2010 (N=323, Missing=0) and by NSRCG respondents who had completed a 
master's degree and were employed as of October 1, 2008. (N=4,326, Missing=0). Items from which these data derive 
differed slightly between the EAPSI Applicant Survey and the NSRCG 2008; thus. Local Government (city, county, school 
district) and State Government (including state colleges/universities) were combined into a single category for both 
groups; and U.S. Federal Government and U.S. MILITARY service, activity duty or Commissioned Corps (e.g., USPHS, NOAA) 
were combined for both groups. 

SOURCE: EAPSI Applicant Survey-Items Die, D2 and D3. NSRCG Survey 2008-Items Bl, Bll and B12. 


5.5 Respondents’ Perceptions of the EAPSI Outcomes 


This last section contributes to a deeper understanding of the outcomes and benefits of the EAPSI 
program by exploring respondents' perceptions of the program. As described below, interviews with 
foreign officials identified a wide range of individuals and groups that benefit from the EAPSI 
program. Specifically, they identified benefits related to EAPSI fellows, their hosts, other individuals, 
and institutions. Survey respondents' perceptions of the benefits for each of these groups are then 
presented. 

5.5.1 Foreign Managers and Agency Officials’ Perceptions of Outcomes 

Responses from the administrators of the EAPSI program at host locations provided useful 
information about the program. Interview respondents saw the program as benefiting multiple 
groups involved in the program, including the fellows, other students in the host lab or institution, 
the host researchers, and more broadly the institutions and countries as a whole. 

For U.S. students participating in the program, most program managers emphasized the increased 
opportunities for collaboration and networking. Additional benefits mentioned included the 
opportunity to gain first-hand research experience and learn how research is conducted in a 
different country. As one respondent explained, “The program enables U.S. students to have 
firsthand research experience in [location] and promotes their collaboration with [location] 
counterparts in the future." 


Abt Associates Inc. 


5. Post-Fellowship Outcomes 72 




Evaluation of the East Asia and Pacific Summer Institutes Program: Final Report 


Respondents also described benefits enjoyed by host country students who interacted with EAPSI 
fellows. The program manager in one location felt that these students benefited from the presence 
of the fellow even more than the host researcher, as they were more directly exposed to new ideas 
and research approaches. An opportunity for the host country students to improve English 
proficiency was also noted by one of the managers. 

Program managers and agency officials also identified benefits to hosts and other researchers who 
worked with the fellows. For example, hosting a U.S. student may allow hosts to learn about 
research, graduate training, and scientific developments in the U.S., and therefore gain a broader, 
international perspective on their research. EAPSI also allows hosts to establish relationships with 
U.S. advisors, which may serve to promote additional international research collaboration. As one 
respondent noted, "[EAPSI] helps to strengthen the link between the [location] hosts and the U.S. 
supervisors. Many times, the U.S. supervisor will come during the program. That relationship 
between senior researchers is initiated or strengthened. The host will subsequently go to the U.S. to 
work with the student or the supervisor." 

Additionally, respondents discussed the benefits that institutions, and the countries as a whole, 
accrue by participating in the EAPSI program. Agency officials felt that participating in EAPSI allows 
them to cultivate good relationships with NSF and the U.S. One respondent commented, "The U.S. is 
an important bilateral partner, so increasing the relationship is great.... We also understand we need 
to be well connected internationally, so this is another program that helps us with this. This program 
supports [location's] international engagement." Further, some agency officials noted that 
participation in EAPSI facilitates research, which furthers the goals of the federal agency. 

Program managers also described broader impacts. For example, they felt that the opportunity for 
increased collaboration, and the resulting exchange of ideas and information, may ultimately spur 
the development of international partnerships and improve their own educational programs and 
research agendas. For example, one respondent explained, "...having these foreign students come 
from different places with their own different perspectives is good for [location] in the sense that 
they can learn about how the U.S. educational system influences their students. They learn from 
students themselves, and this collaboration allows [location] to improve its own programs and get 
new ideas." 

5.5.2 Survey Responders’ Perceived Outcomes 

These outcomes related to fellows, hosts, advisors and others were investigated through specific 
questions in the surveys. These survey findings are presented below. 

5.5.3 Outcomes for Fellows 

The surveys probed specific outcomes for fellows in the areas of educational and career 
opportunities, professional outcomes, and personal outcomes. As described below, the EAPSI 
program is perceived to have positive outcomes for fellows and to help prepare fellows for a variety 
of career opportunities and responsibilities. 
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Education and Career Opportunities 

Overwhelmingly, fellows, hosts, and advisors perceive 



I am sure that my participation i 
the EAPSI program helped make 
my postdoctoral fellowship 
applications more impressive. It 
was great to have a letter of 
recommendation from an 
international leader in my field. 
Also, I wrote about the experience 
in my fellowship applications and 
how the EAPSI program helped me 
understand my commitment to 
scipnce. (EAPSI fellow) 



EAPSI as expanding the options and opportunities of 
participants, without limiting or negatively affecting 
educational and career trajectories. Among fellows 
who were no longer in school, the large majority (78 
percent) responded that EAPSI had made them 


qualified for a broader range of opportunities, while 
less than 1 percent of individuals felt that EAPSI had 
constrained their opportunities. 66 


Less than 1 percent of fellows felt their participation in 
EAPSI had resulted in a lost educational or career 
opportunity. Over 11 percent felt that EAPSI had 




delayed their degree completion. 67 However, V J 

graduation rates were similar for fellows relative to ^ 

unfunded applicants. At the time of the survey, 71 percent of EAPSI applicants had earned the 
degree they were pursuing at the time of application to EAPSI, including 73 percent of EAPSI fellows 
and 68 percent of unfunded applicants. 68 

Negative outcomes or consequences of the program were reported by few individuals: 3 percent 
reported were being more skeptical about international collaboration now than before they had 
participated in EAPSI, and 2 percent reported that competition was introduced between their 
research group and the host. 69 

Similarly, negative outcomes were not commonly reported by advisors; 78 percent reported there 
were no negative outcomes as a result of the fellow's participation in EAPSI. Only 11 percent of 
advisors felt that the fellow's degree completion had been delayed; 7 percent reported that the 
fellow became distracted from important research; 2 percent reported the fellow had left the 
degree program, and 2 percent reported conflicts over intellectual credit for the research conducted 
at the host site. Less than 1 percent reported tension with other members of the home research 
group as a result of the fellowship. 70 

Respondents' descriptions of the outcomes of the program provide a deeper understanding of the 
benefits of the EAPSI program for fellows. For example, the following open-ended responses 
illustrate some of these benefits for EAPSI fellows: 


No exhibit. N=12 to 642, Missing=19 Awarded Applicants. Only fellows who reported that EAPSI did not 
broaden their opportunities (F2=2) responded to item F2b. Source: EAPSI Applicant Survey-Items F2, F2b. 

No exhibit. N=633 to 639, Missing=22 to 28 Awarded Applicants. Source: EAPSI Applicant Survey-Item F6. 

No exhibit. N=1303, Missing=0. Source: EAPSI Applicant Survey-Item A5a. 

No exhibit. N=633 to 639, Missing=22 to 28 Awarded Applicants. Source: EAPSI Applicant Survey-Item F6. 

No exhibit. N=639, Missing=5 Advisors. Responses do not sum to 100 because multiple responses were 
permitted. Source: EAPSI Applicant Survey-Item D4. 
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I learned valuable skills that my host is an expert in and in which I also am now an expert. This 
made me a more competitive candidate in the postdoctoral position I eventually took. (EAPSI 
fellow) 


I believe the student returned in a much stronger intellectual position to pursue his graduate 
studies. The student acquired an international reputation among his peers , which has given him 
an established base/community at international meetings. (Advisor to EAPSI fellow) 


Professional Outcomes 
Surveys gathered information from the various 
respondent groups about the professional outcomes 
for fellows. Fellows identified benefits of participation 
that spanned various professional outcome areas 
(Exhibit 5.10). Among fellows who were no longer in 
graduate school, over half noted that they had made 
valuable connections to researchers outside the U.S. 

(52 percent) and that their participation had made 
them more competitive for jobs (51 percent). Other 
common professional benefits included becoming 
familiar with the scientific enterprise at the host site 
(42 percent) and considering additional professional 
opportunities they had not previously considered (40 percent). 


I believe the EAPSI experience was 
invaluable to my career. It gave me 
a unique opportunity I may not 
otherwise have had. It has shown 
me that international collaboration 
can be very fruitful, especially in 
combining research expertise from 
my home and host labs. (EAPSI 
fellows) 


Exhibit 5.10: Professional Benefits of Participation as Reported by EAPSI Fellows 



Fellows 

1 made valuable connections to researchers outside the U.S. 

51.6% 

EAPSI participation made me more competitive for jobs 1 was interested in 

50.7 

1 became familiar with the scientific enterprise in my EAPSI site 

42.1 

EAPSI participation made me consider professional opportunities 1 would not have 
considered in the past 

40.0 

My work at the host institution opened up new areas of investigation 

34.8 

1 became committed to international research collaboration 

31.7 

My work at the host institution resulted in a substantial advancement in my 
research 

24.6 

My career goals changed from an academic to a non-academic career 

9.1 

My career goals changed from a non-academic to an academic career 

3.5 

1 decided to pursue a graduate degree in a different discipline than the one 1 was 
pursuing when 1 began my EAPSI fellowship 

1.5 

None of the above 

7.0 


NOTES: N=645Missing=16. 

Answered by EAPSI fellows who were no longer enrolled in graduate school as of October 1, 2010. Responses do not 
sum to 100 because multiple responses were permitted. 

SOURCE: EAPSI Applicant Survey-Item F3 

Advisors were asked to rate the success of the program in helping fellows achieve several specific 
outcomes related to research and collaboration (Exhibit 5.11). A majority of the advisors reported 
that the program was successful (highly or somewhat) in helping fellows gain exposure to another 
country's research enterprise (86 percent), advance his/her research agenda (78 percent), establish 
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collaborations with a researcher outside the U.S. (75 percent), learning a technique or approach 
from the host (72 percent), foster long-lasting international collaborations (62 percent), and gaining 
access to resources or materials not widely available in the U.S (58 percent). 


Exhibit 5.11: Advisors' Perceptions of Professional Benefits of EAPSI Program for Fellows 


In your view, how successful was 
the program in helping the fellow 

Highly 

Somewhat 

Somewhat 

Highly 

Do not 

achieve the following? 

Successful 

Successful 

Unsuccessful 

Unsuccessful 

Recall 

To gain exposure to another 
country's research enterprise 

67.1% 

18.8% 

0.9% 

9.4% 

3.8% 

To advance his/her research 
agenda 

50.2 

27.4 

7.2 

11.4 

3.8 

To establish a collaboration with a 
researcher outside the U.S. 

45.3 

29.6 

8.6 

10.5 

6.0 

To learn a technique or an 
approach from the host scientist 

39.5 

33.2 

8.8 

8.0 

10.4 

To gain access to resources or 
materials not widely available in 
the U.S. 

29.8 

28.5 

11.0 

9.7 

21.0 

To foster a long-lasting 
international collaboration 

26.1 

36.2 

15.9 

11.7 

10.1 


NOTES: N ranges from 628 to 639, Missing ranges from 5 to 16. 
SOURCE: EAPSI Advisor Survey-Item C2 


Professional benefits specific to the relationships and research collaborations established during the 
fellowship were also investigated. Post-fellowship, 20 percent of fellows reported having 
collaborated on a research project with their former host, and an additional 60 percent had 
communicated with their host; only 21 percent had not communicated with their host. 71 Among 
those who reported collaborating with their host, 54 percent had collaborated within the past year, 
while it had been 3 years or more since the collaboration for 30 percent. 72 The collaborations most 
commonly consisted of co-authoring papers (72 percent) and exchanging ideas, data, research 
results or tools (67 percent). 73 

The following open-ended responses illustrate some of the professional benefits for EAPSI fellows: 

My EAPSI experience has helped prepare me for working in an international environment. For 
example, I am more able to lead a group of engineers with backgrounds from all over the world 
because of my experiences in [country]. Also, I am more prepared to present my research in 
other countries because of my experiences in [country], (EAPSI fellow) 


No exhibit. N=643, Missing=18 Awarded Applicants. Source: EAPSI Applicant Survey-Item FI. 

No exhibit. N=146, Missing=0 Awarded Applicants. Only fellows who collaborated with their host (Fl=l) 
answered this item. Source: EAPSI Applicant Survey, Item Fib. 

No exhibit. N=146, Missing=0. Only fellows who collaborated with their host (Fl=l) answered this item. 
Source: EAPSI Applicant Survey, Item Fla. 
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Participation in EAPSi gave me insight into how research was conducted in other countries. I was 
exposed to different research perspectives and processes. This exposure helped me to better 
understand the benefits and drawbacks of my existing research strategy. (EAPSI fellow) 

The project completed during the stay at my host institution became a publication in the 
[journal], the foundation for my successful [program] application, and the beginning of an 
ongoing collaboration (which was later further supported by a [program] Postdoctoral 
Fellowship). ('EAPSI fellow) 

[The fellow] learned a specific technique from her host that would have been difficult to learn 
elsewhere, and this has helped increase the collaborative relationship - so much so that a PhD 
from the host lab will join my lab as an NSF-funded postdoc. (Advisor to EAPSI fellow) 

Advisors also identified the interactions they had with fellows subsequent to their fellowship 
(Exhibit 5.12). Most commonly, advisors helped the fellow integrate the fellowship research into 
their research in the U.S. (63 percent), advised the fellow on work that continued what was begun 
during the EAPSI fellowship (57 percent), and assisted the fellow in preparing a presentation or 
publication on research conducted while at the host institution (51 percent). 


Exhibit 5.12: Advisors' Interactions with Fellows upon Return to the U.S. 



Percent 

Upon the fellow's return to the U.S., which of the following were true? 

(N=641) 

Helped the fellow integrate the research conducted at the host institution into their 
research conducted in the U.S. 

62.7 

Advised the fellow as he/she continues work begun during the EAPSI summer 

56.5 

Assisted the fellow in preparing a presentation and/or publication on research 
conducted at the host institution 

51.0 

Helped the fellow use the EAPSI experience to benefit his/her subsequent job search 

42.3 

Provided the fellow financial support to continue work on the research conducted 
duringthe EAPSI summer 

41.3 

Helped the fellow maintain a professional collaboration developed at the host 
institution 

32.1 

Helped the fellow to pursue additional international research opportunities 

28.7 

Other 

3.1 

None of the above 

12.9 


NOTES: Missing=3. 

SOURCE: EAPSI Advisor Survey-Item Cl 


Personal Outcomes Reported by Fellows 

EAPSI is designed to provide fellows with an orientation to the society, culture and language of the 
host site. Indeed, among the personal benefits noted were comfort with the traditions and culture 
of the host location and personal connections (Exhibit 5.13). 
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Exhibit 5.13: Personal Benefits of Participation as Reported by EAPSI Fellows 

! Percent j 

1 became comfortable with the traditions and culture of my host site 

79.1 

1 made personal connections in the places 1 visited during the fellowship 

77.9 

1 gained proficiency in another language 

27.2 

1 decided to live outside the United States 

17.7 

None of the above 

3.7 


NOTES: N=645, Missing=16. 

Notes: Answered by EAPSI fellows who were no longer enrolled in graduate school as of October 1, 2010. Responses do 
not sum to 100 because multiple responses were permitted. 

SOURCE: EAPSI Applicant Survey-Item F5 

Similarly, 64 percent of advisors felt that the program was highly successful in allowing the fellow to 
become familiar with the culture 
and traditions of another country, 
and an additional 19.1 percent 
thought the program was 
successful in this area. 74 

5.5.4 Outcomes for Hosts 

The EAPSI program also provided 
hosts with unique opportunities 
and occasions for professional 
gains. Hosting the EAPSI fellow 
was the first opportunity in which 38 percent of hosts worked with a U.S. graduate student . ,b 

Hosts were asked whether they perceived any benefits accruing to themselves as a result of their 
participation in EAPSI (Exhibit 5.14). Most hosts identified some benefit; only 11 percent perceived 
no personal benefits from their participation. The most common benefits selected by respondents 
included enhanced interest in collaborating with U.S. researchers (31 percent), established or 
renewed collaborations with other U.S. researchers (30 percent), and published papers based on the 
collaborative research conducted (25 percent). 


The EAPSI program allowed the establishment of an on- 
going collaboration between my lab and that of the U.S. 
graduate mentor. A peer-reviewed manuscript is under 
review at the moment from the work completed by the 
fellow (and the subsequent work by new members of my 
lab). I am organizing a visit to the U.S. lab by one of my 
current graduate students as part of this collaboration. 
(EAPSI host) 


No exhibit. N=640, Missing=4 Advisors. Source: EAPSI Advisor Survey-Item C2. 
No exhibit. N=658, Missing=7 Hosts. Source: EAPSI Host Survey-Item E2. 
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Exhibit 5.14: Hosts' Perceptions of Benefits from Participation in EAPSI Program 

I Percent 

Participating in the program enhanced my interest in collaborating with U.S. researchers 

30.7 

1 established or renewed a collaboration with other U.S. researcher(s) 

29.6 

1 published research papers based on the collaborative work with this EAPSI fellow 

25.4 

1 learned new methodological/analytical techniques or theoretical approaches 

20.8 

1 became more familiar with the research enterprise of the United States 

20.8 

1 gave one or more presentations based on the collaborative work 

19.1 

Hosting a EAPSI fellow enhanced the recognition of my work by peers 

14.0 

Improved my English language skills 

11.4 

Participating in the program helped advance my career 

8.8 

Participating in the program helped me recruit other graduate students or postdoctoral 
fellows 

8.7 

Participation in the program changed the direction of some research projects in my group 

7.1 

1 obtained access to resources not easily available at my institution/location 

4.7 

1 obtained funding based on the collaborative work 

4.3 

Other benefits 

8.1 

None of the above 

11.4 


NOTES: N=658, Missing=7. Results do not sum to 100 percent because multiple choices were permitted. 
SOURCE: EAPSI Host Survey-Item E2 


Most hosts reported follow-up communications with the fellow after the fellowship. Specifically, 29 
percent of hosts reported that they had collaborated further with fellows after the summer 
fellowship, 48 percent had communicated, but not collaborated with the fellows. 76 Among those 
who reported collaboration, 66 percent reported it had been within the past year, while 13 percent 
reported the collaboration was three or more years ago. 77 The collaborations most often included 
the exchange of ideas, data, research results or tools (77 percent) and co-authoring a research paper 
(64 percent). 78 On average, hosts reported one research publication resulting from their 
collaboration with the fellow. 79 Approximately one-quarter of hosts also reported collaborating with 
the EAPSI fellow's colleagues, for example former faculty advisors (27 percent). 80 

Open-ended responses also described benefits to the hosts and researchers in the host labs. 

I spent a great deal of time educating the six graduate students in my host laboratory about our 
model organism, about how to apply for post-doc positions in the United States, about working 
in scientific laboratories in the United States, and about the culture of the United States. (EAPSI 
fellow) 


No exhibit. N=654, Missing=ll Hosts. Source: EAPSI Host Survey-Item Dl. 

No exhibit. N=192, Missing=0 Hosts. Only hosts who collaborated with the fellow (Dl=l) answered this 
item. Source: EAPSI Host Survey, Item Dla. 

No exhibit. N=192, Missing=0 Hosts. Only hosts who collaborated with the fellow (Dl=l) answered this 
item. Source: EAPSI Host Survey, Item Dlb. 

No exhibit. N=565, Missing=100 Hosts. Source: EAPSI Host Survey-Item D4. 

No exhibit. N=661, Missing=4 Hosts. Source: EAPSI Host Survey-Item D3a. 
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It's nice to have a foreigner in the lab so that local graduate students can get a chance to 
interact with. Because of similar field that we studied, some scientific discussion and 
conversation were good for me as well as students in the lab. (EAPSI host) 

5.5.5 Broader Reach 

The evaluation also explored areas in which the influence of the program might extend more 
broadly to individuals associated with the direct participants in the program. Some fellows identified 
ways they perceived the experiences extended beyond the direct participants (Exhibit 5.15). Most 
commonly, they noted that their peers became interested in international collaborations (46 
percent) and that research methods or ideas learned from the fellowship benefited others at their 
institution (37 percent). 


Exhibit 5.15: Additional Benefits of EAPSI Participation that Extend Beyond Fellows 


As a result of my participation... 

Percent 

My peers became interested in international collaboration 

45.8 

Research methods or ideas that 1 learned benefited others in my institution 

37.0 

Samples that 1 collected or tools that 1 developed benefited others in my institution 

21.2 

Others in my U.S. research group began an international collaboration 

20.3 

None of the above 

19.5 

Researchers that 1 met during my fellowship joined my research group 

6.7 

Other benefits not listed above 

4.8 


NOTES: N=645, Missing=16. Analysis only includes EAPSI fellows who were no longer enrolled in graduate school as of 
October 1, 2010. Results do not sum to 100 percent because multiple choices were permitted. 

Missing=16 (Fellows). 

SOURCE: EAPSI Applicant Survey-Item F4 


EAPSI fellows reported sharing or engaging in 
activities that would extend the benefits of their 
experiences to others. Specifically, 59 percent 
reported sharing resources or tools collected or 
developed during the EAPSI fellowship, and 56 
percent reported teaching colleagues, students, or 
peers research methods they had learned during 
EAPSI fellowship. 81 EAPSI fellows also reported 
engaging in some activities that might promote 
international collaborations among others. 
Specifically, 32 percent hosted researchers or 
professional colleagues from another country at 
their institution, 15 percent established a program 
to foster international collaborations, 12 percent 
have led a delegation of colleagues to visit a 
research laboratory, university, or business in 


I think this program is an incredible 
opportunity for graduate students. It 
gives them the chance to learn about 
another culture, they learn new 
techniques and approaches, they develop 
potential future collaborations, and it 
makes them more competitive for jobs 
and postdocs. Many universities want 
undergraduates to have international 
experiences and I think this program 
prepares future faculty to lead 
international programs. (Advisor of 
EAPSI fellow) 


81 No exhibit. N=644-646, Missing=15 to 17 Awarded Applicants. Source: EAPSI Applicant Survey-Item D9. 
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another country, and 10 percent established or served in a leadership role for an international 
professional association. 82 

Another mechanism to promote international collaborations beyond those who were direct 
participants is for EAPSI fellows to mentor other individuals from the U.S. who conducted research 
abroad (Exhibit 5.16).Thirty percent of fellows reported mentoring individuals conducting research 
abroad, most commonly graduate students (20 percent), undergraduate students (13 percent), and 
postdocs (3.4 percent) among others. 


Exhibit 5.16: EAPSI Fellows' Mentoring of Individuals Conducting Research Abroad 



Fellows mentored... 

Percent j 

Graduate students 


20.4 

Undergraduate students 


12.8 

Postdocs 


3.4 

Research scientists 


2.3 

Faculty 


0.9 

Other individuals 


6.0 

Did not mentor any individuals who conducted research abroad 

69.8 


NOTES: N=653, Missing=8. Results include only EAPSI fellows who were no longer enrolled in graduate school as of 
October 1, 2010. Results do not sum to 100 percent because multiple choices were permitted. 

Missing=8 (Fellows). 

SOURCE: EAPSI Applicant Survey-Item D8 

Forty-four percent of advisors felt they were more likely to get involved in international research 
collaboration because of the EAPSI program, while 56 percent felt it had not affected the likelihood 
that they would engage in international collaboration; less than 1 percent stated it made them less 
likely to collaborate. 83 

Advisors also identified benefits that they or their research groups derived as a result of the EAPSI 
fellowship. Most commonly, they reported that the fellow's experience broadened their 
understanding of the research enterprise in the foreign country (60 percent), introduced new 
knowledge, approaches, or skills (57 percent), and raised the research group's visibility abroad (52 
percent) (Exhibit 5.17). 


No exhibit. N=639-642, Missing=19 to 22 Awarded Applicants. Source: EAPSI Applicant Survey-Item DIO. 
No exhibit. N=632, Missing=12 Advisors. Source: EAPSI Advisor Survey-Item E7. 
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Exhibit 5.17: Benefits for Advisors and Their Research Groups 

I Percent | 

The fellow's experience broadened our understanding of research enterprise in the foreign 
country 

59.8 

The fellow introduced new knowledge, approaches, and/or technical skills 

56.7 

The fellow made my group and our research better known abroad 

51.8 

The fellow helped establish or maintain an important collaboration that has benefited my 
research/my groups research 

41.3 

We published papers which resulted from the fellow's participation in EAPSI 

30.5 

The fellow brought back samples or other materials unavailable or not easily accessible in 
the U.S. 

25.0 

The fellow's experience changed the direction of our research or added a new dimension 
to our research 

20.7 

Other 

5.2 

None of the above 

9.5 


NOTES: N=639, Missing=5. Results do not sum to 100 percent because multiple choices were permitted. 
SOURCE: EAPSI Advisor Survey-Item D2 


A large percentage of advisors (88 percent) of advisors reported that they currently collaborate with 
researchers in countries outside the U.S. Among these were the researcher who had hosted their 
former graduate student (40 percent), other researchers at the host location (39 percent) and 
researchers in other countries (92 percent). These collaborations spanned activities including co- 
authoring papers to sending students to work in each other's labs (Exhibit 5.18). 


Exhibit 5.18: Nature of Advisors' Collaborations with Individuals outside U.S. 




Percent With 

Percent With 



Other 

Researcher in 



Researchers 

Other 


Percent With 

in Host 

Foreign 

Collaboration 

Host 

Country 

Countries 

Co-author papers 

34.1 

27.4 

75.3 

Share data or information 

34.1 

31.4 

73.3 

Collaborate on a research project 

28.9 

29.0 

78.7 

We send our graduate students and/or postdoc fellows 
to work in each other's labs/sites 

14.7 

13.8 

41.8 

Co-sponsor professional conferences 

9.5 

12.7 

38.0 

Co-author patent/license applications 

1.6 

1.8 

5.6 


NOTES: N=558, Missing=l. Results do not sum to 100 percent because multiple choices were permitted. Results only 
include advisors whose response to Fl=l (indicating that they currently collaborate with researchers outside the U.S. 
SOURCE: EAPSI Advisor Survey-Item Fib 


These benefits for fellows' U.S. advisors and colleagues were described in open-ended responses: 

She returned with a much broader worldview, global imagination, and an enthusiasm for field 
research that she communicated to other students. (Advisor to EAPSI fellow) 

We developed a strong and continuing collaboration with colleagues that my student worked 
with in [location]. We are still realizing benefits of this collaboration. (Advisor to EAPSI fellow) 
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While in [location, the fellow] gave a couple of talks describing our group which increased our 
visibility in our field of study. The host was very impressed by [the fellow] and subsequently 
contacted me to seek whether I would be willing to send other students through this program. 
(Advisor to EAPSI fellow) 

It helped establish a strong connection with host faculty member, which has endured and led to 
subsequent collaborations. It helped us learn about relevant research in [location] that was not 
widely available in the U.S., and it helped promote our own research to others abroad. (Advisor 
to EAPSI fellow) 

5.5.6 Institutions 

Respondents also reflected on whether EAPSI had 
broader effects that extended to institutions. 

Specifically, hosts were asked whether changes had 
taken place at their institutions as a result of their 
participation as an EAPSI host (Exhibit 5.19). Over 
half did not identify any changes (54 percent). 

Changes that were identified included: 
collaborations helped attract students or 
researchers to the institution (21 percent), 
colleagues increased collaborations with U.S. 
researchers (18 percent), and the department 
became more supportive of collaboration with U.S. 
researchers (13 percent). 


Exhibit 5.19: Changes at Host Institutions 


1 Percent 

Collaborations with U.S. researchers helped attract students and other researchers to 
my institution 

20.6 

My colleagues increased their own collaborations with U.S. researchers 

18.2 

Administration in my department(or unit) became more supportive of collaboration 
with U.S. researchers 

13.0 

Additional policies, procedures, or structures have been put in place at my institution 
to facilitate international collaboration 

8.7 

Other 

1.8 

None of the above 

53.8 


NOTES: N=664, Missing=7. Results do not sum to 100 percent because multiple choices were permitted. 
SOURCE: EAPSI Host Survey-Item E4 


Among many of my interactions with the host institution, his stay under the support by NSF 
became one of the critical components for the host institution to establish its overseas operation 
office at [U.S. institution], (Advisor to EAPSI fellow) 

Advisors also reported whether they thought that there were changes in their department, as a 
result of the fellow's participation in the EAPSI program, in the 2-year period following the fellow's 
EAPSI experience (Exhibit 5.20.) 


International linkages and fellowships are, 
I think, an increasingly valuable means of 
sharing knowledge, moving research 
forward, and gaining perspective. I also 
think a small country that is isolated like 
[location] can benefit even more strongly 
from such interactions, and thus such 
programs have added value in that regard. 
(EAPSI host) 
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Exhibit 5.20: Changes in Advisors' Departments 

1 Percent | 

Graduate students became more interested in collaboration with international 
researchers 

47.4 

Collaborations with international researchers helped attract students, postdoc fellow, 
and other researchers 

27.7 

Collaborations with international researchers raised the prestige of the department 

21.8 

Graduate students in this department began collaborating with international 
researchers 

17.8 

Faculty in this department became more interested in collaboration with international 
researchers 

17.2 

Collaborations with international researchers helped bring additional funding to the 
department 

12.1 

Faculty in this department began collaborating with international researchers 

11.4 

Administrative policies, procedures, or structures in the department or institution 
were established to facilitate collaboration with international researchers 

4.9 

1 do not recall 

13.1 

Don't know, 1 was no longer at the department 

2.3 

None of the above 

22.5 


NOTES: N=639, Missing=5. Results do not sum to 100 percent because multiple choices were permitted. 
SOURCE: EAPSI Advisor Survey-Item E2 


These benefits were also described in open-ended responses. 

Other faculty and students were exposed to international collaborative research and the value of 
such research collaborations. We have currently applied for additional funding to help continue 
the current project as well as expand the project by integrating it with additional international 
collaborators. (Advisor to EAPSI fellow) 

The fellow's participation in the EAPSI's program increased awareness amongst graduate 
students in the program about the possibilities to pursue such opportunities and the benefits of 
pursuing them. (Advisor to EAPSI fellow) 

As discussed in the concluding chapter, the findings from the evaluation provide evidence that the 
EAPSI program is providing opportunities that align with its program model, and that lead to both 
intended and unintended benefits. 
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6 Conclusions 


The evaluation of EAPSI examined the 
characteristics and motivations of EAPSI 
participants, the opportunities for 
professional and personal growth provided 
through EAPSI, and the professional outcomes 
related to the program. Findings from this 
study provide evidence that EAPSI is 
contributing to NSF's efforts to help develop 
an internationally competitive and globally 
engaged S&E workforce. Specifically, the 
evaluation found evidence that EAPSI is 
meeting its goals to: 

• introduce U.S. graduate students to East 
Asia and Pacific S&E in the context of a 
research setting; and 

• help students initiate scientific 
relationships that will better enable future 
collaboration with foreign counterparts. 


This program ... provides valuable cultural 
and research experiences for graduate 
students. This program is an excellent way 
to facilitate cultural exchange among 
students and good will between 
researchers from different countries. The 
College of Engineering at [U.S. institution] 
is strongly supportive of international 
collaborations among research faculty and 
students, and likes to highlight these 
opportunities for students. While I cannot 
say for sure that my graduate student's 
participation in the EAPSI program was 
directly responsible for the highly 
encouraging environment we have for 
international collaborations and student 
exchange, I have used it as a positive 
example of the benefits that such 
experiences can have for our graduate 
students. (Advisor of EAPSI fellow) 


The evaluation also found evidence of 
ancillary benefits of the program, which include orienting fellows to the host's society, culture, and 
language, and providing opportunities whose benefits extend beyond the EASPI fellows. Below, the 
program's progress toward its stated goals as well as its broader influences are summarized. 


6.1 Introduce Students to East Asia and Pacific Science and Engineering 
Research 


Research experiences that included collaborations with foreign scientists at their host sites were 
central to the fellows' EAPSI program experiences. The majority of EAPSI applicants did not have 
extensive academic and travel experiences outside the U.S. at the time of application. Thus, through 
the research opportunities, EAPSI introduced fellows to international experiences. 

Reflecting the globalization of science, both home departments and host departments were 
generally supportive of international research. Both fellows and unfunded applicants reported being 
attracted to the program because it offered them an opportunity to enhance their skills and 
knowledge as a researcher. The fellowships provided opportunities for fellows to engage in research 
activities independently as well as collaboratively, and fellows reported being satisfied with the 
various aspects of their research and their interactions with their hosts. Overwhelmingly, EAPSI 
fellows would recommend the fellowship to other graduate students seeking an international 
experience. 
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Graduate students apply to EAPSI because they are interested in pursuing an international research 
experience. Several findings suggest that unfunded applicants continue to pursue other avenues for 
their international experiences. Among EAPSI applicants who held a PhD, EAPSI fellows and 
unfunded applicants held a similar number of international postdoctoral fellowships, and EAPSI 
fellows were less likely than unfunded EAPSI applicants to have been employed outside the U.S. 


6.2 Initiate Relationships That Enable Future International Collaborations 


A key goal of the EAPSI program is to help students initiate scientific relationships that will better 
enable future collaboration with foreign scientists. The evaluation provided evidence that the 
fellows initiate scientific collaborations during their fellowships and that fellows and advisors 
perceived the fellowships as expanding the opportunities available to fellows. Over half of the 
fellows indicated that their participation led to valuable connections to researchers outside the U.S., 
and the majority of host and fellows reported follow-up communications or collaborations after the 
fellowship had ended. 

Fellows also continue productive working relationships with individuals outside the U.S. Although 
unfunded applicants, who work with a collaborator in a foreign country, were more likely than 
fellows to characterize their collaborations as ones that involved joint publications or jointly 
developed products, in actual products, fellows reported a higher number of publications and a 
higher proportion of their publications that were co-authored with a foreign collaborator, providing 
evidence that the EAPSI experiences lead to more productive international research collaborations. 


6.3 Orient Fellows to Host Society, Culture, and Language 


Applicants are also drawn to EAPSI in part because it provides an opportunity to learn about the 
society, culture, and language of host sites. Nearly all fellows reported participating in cultural and 
leisure activities while abroad, and a larger majority reported becoming comfortable with the 
traditions and culture of their host site. 

Many fellows who went to locations where English is not a major language reported engaging in 
language training. Nonetheless, some fellows still reported challenges related to communication 
and language, although most hosts did not report challenges related to the lack of familiarity. 


Extend Benefits Beyond EAPSI Fellows 


The EAPSI fellowship experience and benefits also extend to researchers in the U.S. For example, 
during the EAPSI fellowships, about one-third of fellows reported that other researchers from their 
home institution became involved in the research project. They continue to promote international 
collaborations after they return to the U.S.; specifically, the evaluation found that fellows were more 
likely than unfunded applicants to engage in a series of activities to foster international 
collaborations among others. 

Among the reasons that foreign locations gave for becoming involved in EAPSI were to increase 
research collaborations with the U.S. and to establish a research network and ongoing relationships 
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among scientists. Hosts, also, were drawn to EAPSI both by their interest in the research projects of 
the fellows, as well as the benefits they feel might accrue to their research groups in addition to 
themselves. Some pointed to specific benefits such as increased interest in collaborating with U.S. 
researchers and/or new or renewed collaborations with U.S. researchers. 


6.5 Conclusion 


The evaluation demonstrated that individuals derive benefits from the program, both on an 
individual and a collective basis. EASPI fellows and hosts recommend the EAPSI program to their 
colleagues, and EAPSI U.S. advisors would also encourage additional students to participate. EAPSI 
offers graduate students an opportunity to conduct research in foreign locations and to establish 
relationships that they may build on subsequently in their careers. The experiences also extend 
beyond the specific participants and seed additional international S&E research activities. As such, 
EAPSI contributes to NSF's efforts to promote international S&E among early-career scientists and 
engineers. 
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Appendix A: NSF-Generated EAPSI Descriptions 
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Invasive Neighbors Disturb Taiwan's Coasts 


University of Georgia student Virginia Schutte describes how a summer in Taiwan gave her the 
opportunity to get to the bottom of an invasive species problem in mangrove forests there 



Virginia Schutte collecting data in the field on how 
the cordgrass invasion affects marsh fauna. 


July 28, 2011 

When I was offered a chance to study in Taiwan, I was very excited for the opportunity to do environmental 
research in a different part of the world. But, I was also nervous about living in Asia. My only experiences 
with Asian culture before visiting Taiwan came from eating Chinese food and watching people eat starfish on 
"The Amazing Race." 

Would studying in a different environment be worth braving the unknown? 

The answer, of course, is a solid yes. I received an East Asia and Pacific Summer Institutes ( EAPSI ) 
fellowship from the National Science Foundation (NSF) to study in Taiwan in the summer of 2010. EAPSI 
partnered with Taiwan's National Science Council to send 25 graduate students from the United States to 
Taiwan last year to foster international collaborations between the two countries. 

My host was Flwey-Lian FHsieh of Academia Sinica in Taipei, Taiwan, who researches food webs--the energy 
and nutrient linkages between all the organisms in an ecosystem. 

My own doctoral work focuses on forests of mangrove trees that live right on the edge of the ocean. For the 
EAPSI fellowship, my research linked our interests through a study of Taiwanese mangrove food webs that 
are being disrupted by an invasive species. 

I nvasive studies 

Invasive species are important to study because they can alter the way that ecosystems work. The 
invaders disturb predator-prey dynamics, make native species less abundant and reduce native species 
biodiversity-changes that cause economic and environmental damage at sites worldwide. 



In Taiwan, mangroves colonize coastal mudflats. Cordgrass grows in mudflats on the eastern coast of the 
United States, and decades ago, it was imported to China for aquacultural purposes. Cordgrass has since 
spread to Taiwan, where it is thriving despite eradication efforts. 

The invasive species that scientists usually study compete directly with a similar native species, often 
causing the native species to die back. Unlike other invasive species, cordgrass does not replace or directly 
compete with native species in Taiwan--instead it occupies vacant mudflat space next to mangrove forests. 

So I wanted to know: how does an invasive species affect neighboring native ecosystems? 

Studying marshes 

To answer this question, I focused on mudflat food webs. I put crabs and snails that normally eat mangrove 
materials into cages at the edge of the mangrove forest and provided them with food made from mangrove 
trees, cordgrass or both types of plants. The results will tell me whether those marsh animals prefer to eat 
food from mangroves or cordgrass, and how their diet in an invaded marsh will affect their growth and 
survival. 

I also surveyed marshes around Taiwan to determine whether the animals' food preferences affect their 
foraging strategies. For example, if you're a crab living in the mangrove trees but you would rather eat 
cordgrass, will you change where and how you eat so you can get your favorite food? 

I have many samples left to analyze, but I expect my results to show that the effects of an invader can 
reach beyond the borders of the invaded area to affect organisms next door. 

The creatures that eat mangrove materials play an important role in mangrove ecosystems. They link plant 
materials and predators in coastal food webs, and their eating habits can influence the type and location of 
mangroves on mudflats. If marsh animals change the way they eat because of the cordgrass invasion, this 
could trigger changes in mangrove forests. The forests, when healthy, are a valuable source of food and 
provide humans with protection from storms on tropical coastlines. 

Time in Taiwan 

Studying in Taiwan not only advanced my scientific career, it also broadened my view of Asia. The best part 
of my EAPSI experience was that I didn't just sample some interesting food and landmarks the way a tourist 
would. I was introduced to the country as an insider because of the connections I had with my Taiwanese 
lab mates. 

I went to a wedding, learned not to question the wisdom of ancient Chinese medicine, got to see parts of 
Taiwan that are not on tourist maps and discovered what it was like to be a typical Taiwanese student. 
Because of my host advisor and lab mates, Taiwan will forever be a special place for me. 

I am a field ecologist and because I work outside, I can't create the environment I want to study. I have to 
study the environment that is already there. The EAPSI program gave me the opportunity to research a 
globally significant issue in a unique scientific setting. There are only a few places in the world where an 
invasive species has moved in as a neighbor to a native species without replacing it, and Taiwan was an 
ideal setting for a well-organized scientific study. Understanding how invaders can affect nearby ecosystems 
is an important step forward in preserving native communities and mitigating the effects of invaders all over 
the globe. 

EAPSI let me connect with scientists abroad in a two-way exchange of scientific and cultural practices. 
Although I didn't try any starfish, I grew immensely, both professionally and personally, because of my 
experience. 

--Virginia G. W. Schutte, Odum School of Ecology, University of Georgia, vschutte(5)uaa.edu 


This Behind the Scenes article was provided to LiveScience in partnership with the National Science 
Foundation. 
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The Tasmanian Devil in the Wastewater 


Arizona State University grad student Michelle Meighan on her experiences in Tasmania as a 
participant in NSF's East Asia and Pacific Summer I nstitutes program 



Michelle Meighan feeds a kangaroo during her summer in Tasmania in 2009. 


February 3, 2011 

In the months leading up to the summer, a question that I frequently heard was "Tasmania? You're going to 
spend your summer in Africa?" Another popular comment was "You realize it's winter down there, right?" 

Yes, I was fully aware that I was "giving up" my summer (the 115 degrees Fahrenheit oven that is a Phoenix 
summer) for a somewhat colder climate, but no, I was not going to Africa. 

Tasmania, or "Tassie," is a heart-shaped island south of mainland Australia. I was able to study in Tasmania 
through a fellowship provided by the National Science Foundation (NSF). NSF, in conjunction with the 
Australian Academy of Science, funded 20 American students to travel to Australia to facilitate our research 
here in the U.S. through the East Asia and Pacific Summer Institutes (EAPSI) program. 

I worked with Michael Breadmore in the Australian Centre for Research in Separations Sciences (ACROSS) at 
the University of Tasmania. My research combined my dissertation work in counterflow electrophoretic 
separations (which involves analyzing substances based on how they flow through an electromagnetic field) 
with an ongoing electrokinetic supercharging project in Breadmore's lab. 

Electrokinetic supercharging, although it sounds like a trait of a cartoon hero, is simply a method for using 
an electric field to concentrate substances for analysis. 

Whereas my work at Arizona State University allows me to study proteins with a device developed in-house, 
my work with Breadmore took advantage of more powerful commercial instruments to study non-steroidal 
anti-inflammatory drugs (NSAIDs) in wastewater. 

We chose NSAI Ds--the technical term for a class of common, over-the-counter painkillers--because they are 
hard to eliminate from wastewater, which results in their release into the environment. Once in our 
waterways, NSAIDs have the potential to cause adverse health effects in both aquatic and terrestrial 
organisms. 


In Breadmore's lab, a graduate student had recently finished his dissertation work on the electrokinetic 
supercharging technique. He was a tremendous help for my research and for tackling experimental 
difficulties. 

The research results are aiding development of a technique for improving separations analyses, specifically a 
method that involves increasing pressure when chemicals of interest are injected into the electrophoresis 
system. 

My work focused on experimental parameters--such as the various electrolytes and their concentrations, and 
the duration and magnitude of the potential applied--to effectively separate and concentrate various 
NSAIDs. 


While successful, my experience was not only about research--! also "begrudgingly" fulfilled NSF's request 
for cultural experience. 

Australia is an amazing country. I was fortunate enough to sit in on "Question Time" at the House of 
Representatives in the capital, Canberra, where I gained an appreciation of Australian politics. I also visited 
a wildlife reserve where I learned about the facial tumor that is rapidly killing the population of Tasmanian 
devils. 

Thanks to television, I had believed that Tasmanian devils were brown, which is incorrect--they are black! 
Furthermore, while they don't spin in circles, they do make a hellish noise when feeding. 

Of course, my cultural experience would not have been complete had I also not enjoyed a meat pie at an 
Australian Rules "Footie" game. 

Participating in the EAPSI program was a truly remarkable experience. I was able to work with an amazing 
group of diverse colleagues and research a novel technique, all while immersing myself in the Australian 
culture. The opportunity to collaborate with Breadmore in the ACROSS program enabled me to learn a new 
research method, as well as gain a deeper understanding of electrophoretic separations through various 
interactions with researchers in the university's chemistry department. 

Additionally, through a presentation of my dissertation work, I was able to engage in beneficial discussions 
with my Australian colleagues that have afforded me a fresh perspective on my research, applications and 
methodologies. The experience has been invaluable to my development both on a personal and professional 
level ... even if it wasn't in Africa. 

-- Michelle M. Meighan, Department of Chemistry and Biochemistry, Arizona State University 

This Behind the Scenes article was provided to LiveScience in partnership with the National Science 
Foundation. 
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My Research: I Burn Stuff 


U.S. graduate student Abigail Watrous on her research studying energy technologies and seeking 
practical and affordable alternatives to help developing communities reduce pollution 



A pile of coal in Gansu province, China. Coal use is 
exacerbating the country's pollution problem. 


J une 5, 2008 

As a graduate student, explaining what you do for your research isn't always easy. For me, the long 
explanation is that I look at renewable energy technologies for developing communities, specifically options 
for cooking and heating in rural China. The larger purpose is to reduce carbon dioxide (C0 2 ) emissions and 
improve human and planetary health. 

Some people think this is cool, while for others, their eyes glaze over pretty quickly. So, while my brother 
David was visiting me in Beijing this past summer, he came up with a more concise explanation: "You burn 
stuff." 

This is actually almost true. Rather, about 700 million people (that's more than twice the population of the 
United States) living in rural areas of China, burn stuff. The "stuff" they burn can be straw, rice husks, corn 
husks, corn cobs, wood or coal. Everything except coal, that is, everything organic, is characterized as 
biomass, which is considered a renewable resource. 

Coal is non-renewable and, when burned, creates high levels of C0 2 emissions--bad news. Unfortunately, 
biomass, while renewable, can also create harmful emissions and particulate matter, if burned in certain 
ways. 

While farmers in the past used biomass to a great extent, the use of coal in rural areas in China is 
increasing. China is already struggling with significant air pollution, and the increased emissions from rural 
coal use only exacerbate the problem. However, coal is one of the least expensive energy options, so if we'd 
like farmers to use biomass in a sustainable way, we need to make sure that it is equal to or less than the 
price of coal. 

Part of my research looks at trying to find the most practical, effective and affordable way for rural families 
in China to use biomass (for cooking and heating their homes), without having harmful effects on their 
health or the health of our planet. 


I spent last summer in Beijing as a National Science Foundation (NSF) East Asia and Pacific Summer 
Institute (EAPSI) Fellow, and had the great privilege of conducting research atTsinghua University for eight 
weeks. I had planned before coming to Beijing to stay for a full year, so I am continuing my research here 
and will be in China until just after the start of the 2008 Olympics. 

The EAPSI program was a fantastic way to begin my China adventure. I got to meet about thirty other 
graduate students from all over America, all of whom were studying different topics, and we had a blast 
exploring Beijing together, practicing Chinese and talking about our extremely varied research interests. I'm 
continuing to work at Tsinghua, and have loved getting to know the Chinese graduate students in the rural 
energy research group here. I'm thankful for my advisors, the EAPSI program (which gave a great start to 
my year in this amazing country) and my brother, who gave me a whole new way to explain what I do! 

-- Abigail Watrous, Tsinghua University and University of Colorado at Boulder abbv.watrous@qmail.com 
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40 Million Electric Bikes Spark Environmental 
Dilemma in China 


Engineering professor Christopher Cherry reports on his study of the impact of electric bikes in 
China 



This scene from a parking lot in China drives home 
the growing popularity of e-bikes. 


J anuary 24, 2008 

Grace Zhang is like many other Chinese women. She is a middle-aged business owner, mother of a young 
daughter and one of more than 40 million new users of the electric bike, or e-bike, in China. Zhang is 
among China's emerging and rapidly motorizing middle class, riding China's economic growth. She leads a 
busy life between operating an English school, transporting her child and shopping; her day is full of activity. 
Her daily activities require high levels of flexibility and mobility, needs met by her new e-bike. 

Electric bike users have taken Chinese cities by storm, quickly outnumbering cars and in many cities, 
bicycles. 

Electric bikes range in style from traditional pedal bicycles powered by an electric motor to larger electric- 
powered scooters. They are loosely restricted on speed and size, but given the same rights as bicycle users, 
operate in bicycle lanes and do not require driver's licenses, vehicle registration or helmet use. 

Good or bad? 

Proponents would suggest that the e-bike phenomenon is a positive development; after all, e-bikes are 
quiet, non-polluting and provide more mobility than any other mode of transportation. 

Opponents, however, charge that e-bikes are unsafe, increase congestion and indirectly pollute the 
environment through increased power plant emissions and lead pollution from the vehicles' heavy batteries. 
Several cities have attempted to, or successfully, banned electric bikes from roadways, including the mega- 
cities of Beijing and Guangzhou. 


Still, there has been little research on the true impacts of electric bikes in China. 



As a doctoral student in civil and environmental engineering at the University of California, Berkeley, I began 
conducting research, which led to a dissertation, on quantifying the impacts of electric bikes in China. I 
participated in the National Science Foundation's inaugural East Asia and Pacific Summer Institute (EAPSI) in 
China during the summer of 2005, and with support from the Volvo Foundation, I used this experience to 
examine several of the contentious issues surrounding electric bikes, including their seemingly negative 
impacts on the environment and safety, countered by their mobility benefits that allow access to jobs, 
shopping and health care opportunities. 

Filling a niche 

Chinese cities are expanding and becoming more congested as new personal automobiles fill the scarce 
roadway capacity. With a long legacy of bicycle use, Chinese commuters are accustomed to personal 
mobility and short trips. Trips are getting too long for bicycles and public transit services are often incapable 
of serving populations in cities with disorganized urban development patterns while competing with cars for 
road space. 

Electric bikes have filled the niche, providing high levels of personal mobility at a fraction of the cost of a car 
or even public transit. 

I found that electric bikes travel about 35 percent faster than bicycles and have a much larger range. In the 
city of Kunming, an electric bike can provide access to 60 percent more jobs within 20 minutes than a 
traditional bicycle. Compared to a 30-40 minute bus ride, an electric bike rider can access three to six times 
the number of jobs. 

While this increase in mobility is remarkable, this mobility does come at a cost--namely increased lead 
pollution from battery use. 

The environmental price 

Electric bikes use one car-sized lead acid battery per year. Each battery represents 30-40 percent of its lead 
content emitted to the environment in the production processes, resulting in about 3 kilograms of lead 
emitted per battery produced. When scaled up the 40 million electric bikes currently on the roads, this is an 
astonishing amount of lead emitted into the environment. 

This negative environmental impact is countered by other environmental benefits compared to most modes, 
including vastly reduced energy use and greenhouse gas emissions. 

Ultimately, the success or failure of electric bikes as a sustainable mode of transportation should be 
evaluated in the context of the extent to which they displace automobiles. The e-bikes certainly have fewer 
negative impacts than personal automobiles, but currently displace mostly bus and bicycle users and only a 
small number of car users. 

As China motorizes, will electric bikes displace would-be car users or simply provide a stepping stone to full 
blown auto ownership? They will likely lead to both outcomes. To the extent that electric bike battery 
technology and production processes improve, electric bikes provide some of the highest mobility and access 
to an urban area with some of the lowest negative impacts to the transportation system or the environment. 

-- Christopher Cherry, (865) 974-7710 cherrv(a)utk.edu , Assistant Professor of Civil and Environmental 
Engineering at the University of Tennessee-Knoxville 

This Behind the Scenes article was provided to LiveScience in partnership with the National Science 
Foundation. 
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At the Crossroads of Stem Cells and Computer 
Science 


A Rutgers University graduate student takes readers on a journey from Piscataway, New Jersey, 
to Hsinchu, Taiwan, and shares some experiences with East-West collaboration, stem cell sorting 
and computer science 



Mike Wininger at the Industrial Technology Research Institute, Hsinchu 
Xian, Taiwan. 


March 20, 2008 

My blonde hair, blue eyes and size 15 running shoes weren't the only reasons for which I was a unique 
attraction at the Industrial Technology Research Institute in Hsinchu Xian, Taiwan. My background in physics 
and mathematics meant that I was the only member of Dr. Wannhsin Chen's Stem Cell Engineering Group 
who had no experience culturing human embryonic stem cells. 

I had come to the institute's bioengineering division as an East Asian Pacific Summer Institute summer 
fellow, having come from Rutgers University in New Jersey where I'm a Ph.D. student and fellow of the 
National Science Foundation's (NSF) I ntegratively Engineered Biointerfaces program. I had received cutting- 
edge instruction for the culturing of cells, but my experience was limited to very hardy cell lines from 
epithelial tissues (surface cells), and my approach to cultures was more engineering than biomedical. 

However, in Taiwan, my mentors were all biologists and would only mentor me as a student of biology. They 
were very forgiving of my utter lack of Chinese language skills, but they were much more insistent that I 
speak "biolog-ese." Nevertheless, the work was everything that I had been trained to do at the university-- 
to be able to interface with the researchers in the basic sciences, and to synthesize the experience within 
the universe of biomedical engineering. 

Part of the magic of modern tissue research is just how deftly an experienced biologist can look at an image 
of cells in a dish and know immediately which cells are healthy or not, differentiated or not. These 
qualitative--and in both labs, uncannily accurate--assessments not only confound the former 
physicist/entry-level culturist, but also place a concrete limit on the number of cells that can be evaluated in 
a day, and require valuable human resources--resources that periodically become susceptible to 
inaccuracies, affected by factors such as long days, nearing lunch hours or the hustle-and-bustle of a busy 
lab environment. 


Merging statistical pattern recognition and computer vision to cell imaging, I am addressing the problem of 
cell sorting with novel quantitative methods. By processing hundreds of cell images and extracting salient 
parameters of cell size, morphology and structure, I seek to rigorously and objectively determine a stem 
cell's status. Experimental purity dictates that I devise my algorithm "blinded," i.e., on cell images 
generated by my mentors without my involvement, which is probably for the better: their cells were 
markedly more photogenic than mine (the only trustworthy qualitative statement this amateur can make!). 

When I am finished, a computer will be able to acquire, save, filter and scan cell images, and process them 
for an online categorization, cell-by-cell (freeing the biologists to spend their workday innovating). The idea 
is either to automate the cell sorting process, increasing the through-put of cell passage, or to complement 
the biologist's subjective assessments. These principles underlie a number of fields of biomedical study, 
including automated diagnostics, pharmoaco-theraputics and even forensics. 

Biology and math weren't the only things I learned during my tenure in Taiwan. I learned a lot about myself, 
about academia outside of the United States and about the Taiwanese people. For many of the summer 
fellows, it was our first time out of the country. But the training staff in Taiwan made us feel at home from 
the minute we arrived. 

I saw the experience as an opportunity to reinvent myself as a scientist. In my round trip from Piscataway, 
New J ersey, to Hsinchu, Taiwan, I departed from my hometown of biomedical engineering and landed at the 
crossroads of stem cell biology and computer science. 

-- Mike Wininger, Rutgers University wininqer@eden. rutqers.edu 

Editor's note: This graduate student has been funded as a NSF Integrative Graduate Education and 
Research trainee (IGERT) and received funding through the East Asia and Pacific Summer Institutes for U.S. 
Graduate Students (EAPSI). The experience described is integral to the purpose of helping graduate 
students be globally aware and competitive. 

This Behind the Scenes article was provided to LiveScience in partnership with the National Science 
Foundation. 
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Appendix B: Non-Response Bias Study 


This appendix describes the non-response bias study conducted for the applicant survey. First, it 
describes the two types of missing data. Next, it describes the analytic approach for addressing each 
type of non-response. 


Types of Missing Data 


There are two types of missing data that can arise in a survey, even after repeated attempts to 
collect data: (1) unit non-response, and (2) item non-response. The approach to dealing with each of 
these in this study is described below. 

Unit Non-Response 

Unit non-response occurs when an entire data instrument is not received from a potential 
respondent. Because non-response was greater than 10 percent, available data were used to 
explore whether results might be affected by non-response bias. Large differences in the response 
rates for subgroups could indicate that potential biases may exist 1 . For example, if the response rate 
from women was very low, and women were less likely to belong to the treatment group, then any 
observed difference in the outcomes between the treatment and comparison groups could 
potentially result in a biased estimate of the impact of the treatment. Exhibit B.l shows the 
variables used in this analysis. Exhibits B.2 to B.4 shows the response rates by groups suggesting that 
there were differences in response rates by subgroups. 


Note that a large non-response rate does not necessarily create bias. For example, if the non-respondents 
were similar across the treatment and comparison group, then the impact estimate would not be biased 
necessarily; rather, any effect of the program could not be generalized to the non-respondents (i.e. it 
would create an external validity problem but not necessarily an internal validity issue). 
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Exhibit B.l: Variables Used 

in Non-Response Analysis 


Variable 

Variable Name 

Values 

N (%) 

Gender 

Cr_nrbias_gender 

0=Male 

l=Female 

1187 (57.23) 
887 (42.77) 

Under-Represented Minority 

Cr_nrbias_URM 

0=Non-Minority 

l=Minority 

2=Missing 

1494 (72.03) 
234 (11.28) 
346 (16.68) 

Disability Status 

Cr_nrbias_hdcap 

0=Non-Disabied 

l=Disabled 

2=Missing 

1602 (77.24) 
17 ( 0.82) 
455 (21.94) 

Cohort 

nYear_App_group 

1=2000-2005 

2=2006+ 

841 (40.55) 
1233 (59.45) 

Award Status 

Nfinalawdfin 

l=Awardee 

0=Decline 

1283 (61.86) 
791 (38.14) 

Proposal Score 

Finalscore 

1-5 

Min =0.00 2 
Max = 5.00 
Mean = 3.69 
Std = 0.86 
N =2071 


Examining Response Rates Overall 


Exhibit B.2: Response Rates by Subgroups of Interest. 


Characteristic 


% Responding (n) 


p-value 

Gender 

Males 

Females 

Missing 

0.3173 


62.01 (736) 

64.15 (569) 

- 


Under- 

Non-URM 

URM 

Missing 


Represented 

Minority 

74.36 (1111) 

69.66 (163) 

8.96 (31) 

<0.0001** 

Disability Status 

Non-Disabled 

78.78 (1262) 

Disabled 

52.94(9) 

Missing 

7.47 (34) 

<0.000l" 

Award Status 

Non-Awardee 

46.14 (365) 

Awardee 

73.11 (938) 

Missing 
46.14 (365) 

<0.0001** 

Application 

- 

2000-2005 

2006+ 

0.0319* 

Cohort 


60.17 (506) 

64.80 (799) 

Average Proposal 
Score 

Non-Responders 

3.4143 

Responders 

3.8573 

Missing 

<.000l” T 


*p<0.05, ** p <0.01 , *** p<0.001 

NOTES: Unless otherwise specified p-values are from a 2X2 chi-square test of the null hypothesis of no association 
between participation and the characteristic of interest (1 degree of freedom). 

T p-value is from the Student’s T-Test 


The dummy imputation method was used to impute for 3 missing proposal scores. The missing cases were set to a 
constant (0) and added a missing data flag" to the propensity to respond model. The coefficient on the dummy 
variable measures how far off the imputation was from what would be expected based on the non-missing values; it 
was assumed the missing data is MCAR. 
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Examining Response Rates by Award Status 



Exhibit B.3: Response Rates by Subgroups of Interest Controlling for Award Status. 



% Responding 


p-values 

Characteristic 

(n) 

2X2 Chi Square 

Award Status CM FI 

Gender 

Non-Awardee 

Male 

43.38(190) 

.0823 



Female 

49.58(175) 

0.1343 

Awardee 

Male 

72.63 (544) 

.6463 


Female 

73.78(394) 


URM 

Non-Awardee 

Non-URM 

56.98(298) 




URM 

56.70(55) 

<0.0001** 



Missing 

7.02 (12) 


<0.0001** 

Awardee 

Non-URM 

83.73 (813) 



URM 

77.37 (106) 

<0.0001** 



Missing 

10.86(19) 



Disabled 

Non-Awardee 

Not Disabled 

63.73 (355) 




Disabled 

25.00 (2) 

<0.0001** F 


Awardee 

Missing 
Not Disabled 

3.54 (8) 
86.70 (906) 


<0.0001** 


Disabled 

66.67 (6) 

<0.0001** 



Missing 

11.35 (26) 



Application Cohort 

Non-Awardee 

2000-2005 

38.74(86) 

0.0091*’ 



2006+ 

49.03 (279) 


Awardee 

2000-2005 

67.69 (419) 

0.016* 



2006+ 

78.16(519) 



Mean Score 





Non- 

Mean Score 



Proposal Score 

Responders 

Responders 


p-value 

Non-Awardees 

2.93 

3.13 


0.0004** 1 

Awardees 

3.98 

4.14 


**T 1 1 

0.0003 ' 

Missing values are assigned their own level. 
*p<0.05, ** p <0.01 , *** p<0.001 




NOTES: Unless otherwise specified p-values are from a 2X2 chi-square test of the null hypothesis of no association 

between participation and the characteristic of interest (1 degree of freedom). 


p-value is from the Fisher's Exact Test 
T p-value is from the Student's T-Test 




u Unequal variances assumption used for Student's T-Test 



The column labeled "Award Status CMH" shows the p-values from Cochran-Mantel-Flaenszel tests of the null 

hypothesis of no common participation effect across award status. 
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Examining Response Rates by Application Cohort 


Exhibit B.4: Response Rates by Subgroups of Interest Controlling for Application Cohort. 


% Responding 

Characteristic (n) 

p-values 

Application Cohort 
2X2 Chi Square CMH 

Gender 


2000-2005 

Male 

61.01 (313) 

0.4744 



Female 

58.54 (192) 

0.3448 

2006+ 

Male 

62.46 (421) 

0.0685 


Female 

67.44 (377) 


URM 

2000-2005 

Non-URM 

77.84 (432) 




URM 

69.41 (59) 

<0.0001** 


2006+ 

Missing 

Non-URM 

6.97 (14) 
72.31 (679) 

<0.0001** 

<0.0001** 


URM 

68.46 (102) 



Missing 

11.72 (17) 



Disabled 

2000-2005 

Not Disabled 

81.68 (477) 




Disabled 

80.00 (4) 

<0.0001** F 



Missing 

9.52 (24) 


<0.0001** 

2006+ 

Not Disabled 

77.01 (784) 



Disabled 

33.33 (4) 

<0.0001** 



Missing 

4.93 (10) 




Award Status 


20001-2005 

Non-Awardee 

38.74 (86) 

<0.0001 


Awardee 

67.69 (419) 

2006+ 

Non-Awardee 

49.03 (279) 

<0.0001 


Awardee 

78.16 (519) 


< 0 . 0001 ** 


Mean Score 
Non- 


Proposal Score Responders Mean Score Responders 

p-value 


2000-2005 3.45 3.88 <0.0001 ' 

2006+ 3.36 3.84 <0.000l” T,U 


Missing values are assigned their own level. 

*p<0.05, ** p <0.01 , *** p<0.001 

NOTES: Unless otherwise specified p-values are from a 2X2 chi-square test of the null hypothesis of no association 
between participation and the characteristic of interest (1 degree of freedom). 

F p-value is from the Fisher's Exact Test 

T p-value is from the Student's T-Test 

u Unequal variances assumption used for Student's T-Test 

The column labeled "Application Cohort CMH" shows the p-values from Cochran-Mantel-Haenszel tests of the null 
hypothesis of no common participation effect across application cohort. 


To address the potential for bias, the probability of a person responding to the survey both for 
responding and non-responding individuals was estimated as a function of baseline characteristics 
that were available for both types of individuals (e.g. proposal score, cohort year, gender), and 
created weighting classes for adjusting the weights of responding individuals to alleviate the bias 
due to non-response. Steps 1-4 described below were taken to accomplish this task. 
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Estimating Probability of Response 


Step 1 : Fit Models 

Logistic regression models were fit to estimate the probability of a person responding to the survey. 
The response (dependent) variable is a dummy variable that took the value " 1 " for responding 
applicants and took the value "0" for non-responding applicants. The explanatory (independent) 
variables are the variables described in Exhibit B.l. Models also included all two-way interaction 
terms of award status/application cohort with the other variables described in Exhibit B.l. 

Several models were fit to identify the set of explanatory variables that have statistically significant 
associations with the dependent variable (p<0.20 criterion) after controlling for other statistically 
significant control variables. This was accomplished by using backwards elimination with forward 
checking. 3 In this method, all of the explanatory variables are entered as predictors in the logistic 
regression model. The explanatory variable with the largest non-significant value is dropped from 
the subsequent model. This step is repeated until the only explanatory variables that remain in the 
model are those that meet the p<0.20 criterion. In the forwards checking step, each of the 
previously eliminated control variables is checked by adding each one to the model with only the 
significant predictors. In this step, each variable has a chance to get back into the model. The final 


Backwards elimination methods are attractive from the point of view that they are often used and 
familiar. But use of this method with the conventional p<0.05 criterion has been criticized from the point 
of view that the selection criteria tend to favor covariates with strong relationships to the outcome, but 
may omit important confounders (i.e., variables that have a weaker relationship to the outcome, but have 
a strong relationship to the predictor variable of interest). Maldonado and Greenland (1993) evaluated a 
backwards elimination strategy and a change-in-estimate strategy using simulated data from a poisson 
regression model. They found that the p-value based method performed adequately when the alpha 
levels were higher than conventional levels (0.20 or more), and found that the change-in-estimate 
strategy performed adequately when the cut point was set to 10 percent. However, their data, generated 
from a poisson model, and their analysis model, with only a single covariate in addition to the key 
exposure variable, are very different than the models anticipated for the current purpose. 

Budtz-Jorgensen et al. (2001) compared several covariate selection strategies including backwards 
elimination and change-in-estimate. They looked at the backwards elimination strategy with three p-value 
cut-off levels, 0.05, 0.10, and 0.20, and, following the recommendation of Maldonado and Greenland 
(1993) used a 10% criterion for the change-in-estimate method. They found that, although the change-in- 
estimate strategy did an adequate job of identifying confounders and keeping them in the model, it 
sometimes threw out variables that were correlated with the outcome, but were not confounders. 
Therefore, this method threw out variables that, if retained, would have reduced the residual error and 
reduced the standard error of the exposure coefficient (thus increasing the power to detect exposure 
effects - exposure effect is analogous to the key predictor of interest). Although they found that 
backwards elimination with a p<0.05 criterion was un-suited for confounder identification, they found 
that when the p-value criterion was set to p<0.20, backwards elimination strategy resulted in a reduction 
of residual error variance and did not throw out important confounders. They recommended the 
backwards elimination strategy with a p<0.20 criterion over the change-in-estimate strategy. 

Maldonado, G., Greenland, S. 1993. Simulation study of confounder-selection strategies. American Journal 
of Epidemiology 138(11), 923-936 

Budtz-Jorgensen, E., Keilding, N., Grandjean, P., Weihe, P.,and White, R. 2001. Confounder identification in 
environmental epidemiology. Assessment of health effects of prenatal mercury exposure. Downloaded 
from http://www.pubhealth.ku.dk/bsa/research-reports/paper_ms.ps 
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model indicated that the probability of being a responding applicant was related to: applicant's 
proposal score (final score), award status, award cohort (cohort), under-represented minority status 
(URM), and disability status. The results from the final model are summarized in Exhibit B.5. 

Exhibit B.5: Summary of Final Model Results 


Type III Analysis of Effects 


Variable 


DF 

Wald Chi-Square 


Pr > Chisq 

Final Score 


1 

1.5846 


0.2081 

Score Imputation 


1 

0.0006 


0.9812 

Award Status 


1 

12.5015 


0.0004’"’" 

Cohort 


1 

1.6889 


0.1938 

URM 


2 

12.0256 


0.0024’"’" 

Disability status 


2 

8.0309 


0.0180* 

Award Status*Cohort 


1 

3.7972 


0.0513 

Final Score*Cohort 


1 

4.2091 


0.0402* 

Cohort ’"URM 


2 

3.4074 


0.1820 

Cohort*Disability Status 


2 

3.7234 


0.1554 

The Logistic Procedure 
Analysis of Maximum Likelihood Estimates 


DF 



Wald Chi- 


Parameter 

Estimate 

Standard Error 

Square 

Pr > ChiSq 

Final Score 

1 

-0.3990 

0.3169 

1.5846 

0.2081 

Final Score Imputation 

1 

-10.7278 

454.9 

0.0006 

0.9812 

Award Status 

1 

2.1879 

0.6188 

12.5015 

0.0004** 

Cohort Status 

1 

-0.9899 

0.7618 

1.6889 

0.1938 

Non-URM 

1 

2.8744 

0.8342 

11.8715 

0.0006** 

URM 

1 

2.6899 

0.9522 

7.9806 

0.0047** 

Not Disabled 

1 

1.9765 

0.7009 

7.9521 

0.0048** 

Disabled 

1 

2.3449 

2.4223 

0.9371 

0.3330 

Award*Cohort 

1 

-0.6775 

0.3477 

3.7972 

0.0513 

Final Score*Cohort 

1 

0.3777 

0.1841 

4.2091 

0.0402* 

Cohort ’"Non-URM 

1 

-0.9657 

0.5253 

3.3790 

0.0660 

Cohort*URM 

1 

-0.7897 

0.5967 

1.7518 

0.1856 

Cohort*Not Disabled 

1 

0.8437 

0.4708 

3.2115 

0.0731 

Cohort ’"Disabled 

1 

-0.1734 

1.3712 

0.0160 

0.8994 


*p<.05, ** p <0.01 , *** p<.001 


Step 2: Use Model Results to Calculate Response Propensities 

In Step 2, parameter estimates obtained from the fitted model were used to calculate the predicted 
probability that an applicant will respond to the survey. The logistic regression model is represented 
as: 


log(- 


1-7T ; 



71 

where ' is the probability that applicant i is a responding applicant, and the summation is over the 
k predictor variables in the final model. The predicted probabilities were obtained by solving the 
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TC ■ 

previous equation for 1 , and substituting the parameter estimates from the fitted model in place 
of the parameters. The solution for the predicted probability for applicant i is given by: 


exp (A,+Z/U 

7T- = * * ^ 

1 + exp (Po + ^Pu) 

k 

K 

Each applicant's predicted probability of response ( ' ) is called its "response propensity". 
Applicants with similar response propensities have similar characteristics. In particular, they are 
similar on the characteristics that are most related to the probability of response. 

Step 3: Group Applicants with Similar Response Propensities into Weighting Classes 

In this step, applicants with similar response propensities were grouped into weighting classes. The 
weights of responding applicants within a class were inflated so that the responding applicants 
within the class represent the population that both the responding and non-responding applicants 
within the class were originally sampled to represent. Exhibit B.6 shows the distribution of response 
propensities for the applicant sample. 

Exhibit B.6: Distributions of Propensity Scores (All Applicants) 



Weighting classes were formed to ensure that all applicants within a class fell within a narrow range 
of propensity scores. The boundaries for the weighting classes were determined by creating 
approximately equal-interval propensity score groupings. The top and the bottom of each 
propensity interval differed by .024 to ~.35 points. The resulting five classes corresponded to 
propensities in the ranges of "0-35, 35-68, 68-84, 84-88, and 88-91 percent probability of 
response. Exhibit B.7 shows the frequency and percent of applicants that fell within each of the five 
weighting classes. 
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Exhibit B.7: Number and Percent of Applicants in Each of Five Weighting Classes (Results for All 
Applicants in the Sample) 


Weighting Class 

Frequency 

Percent 

Cumulative 

Frequency 

Cumulative 

Percent 

l)Propensities ~0-.35 

468 

22.57 

468 

22.57 

2)Propensities .35-. 68 

417 

20.11 

885 

42.67 

3)Propensities .68-. 84 

301 

14.51 

1186 

57.18 

4)Propensities .84-. 88 

464 

22.37 

1650 

79.56 

5)Propensities .88-. 91 

424 

20.44 

2074 

100.00 


Step 4. Within Weighting Class, Inflate Weights of Responding Applicants to Sum to 
Population Total 

Within each weighting class, the weights of all applicants (both responders and non-responders) 
were summed. Next, the weights of just the responding applicants were summed. Then, within each 
weighting class, new, adjusted weights of responding applicants were calculated by multiplying the 
initial weights of 1 by a factor equal to the ratio of the sum of the weights of all applicants to the 
sum of the weights of the responding applicants. The adjusted weight for the / th applicants in the / h 
weighting class is represented symbolically by: 


I 


w 


kj 


a d\ . A: g responders & nonresponders 

W- = W. . * 

rv ii rv ii 

2 > y 

i g responders 

where w.. is the initial sampling weight for the / th applicants in the / h weighting class, the summation 


in the numerator is over all k applicants in the set of responders and non-responders within 
weighting class j, the summation in the denominator is over all / applicants in the set of responders 
in weighting class j, and there are j = 1,..., 5 weighting classes. The new, adjusted sampling weights 
sum to the population total number of applicants. This result can be written symbolically as: 


X X< i; =Z X w kj 

j i g responders j k g responders & nonresponders 

Exhibit B.8 shows that the weights of the 2,074 applicants in the sample sum to the total number of 
applicants in the target population (N=2,074). Exhibit B.9 shows the weights summed within each of 
the five weighting classes. The sum shown for the y' th weighting class ( / = 1,...,5) corresponds to the 
term: 


X w kj 

k g responders & nonresponders 


For example, the applicant weights of the 468 applicants in the first weighting class sum to: 

X w *, = 468 - 

k g responders & nonresponders 
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Exhibit B.8: Size of Target Population (Sum of Initial Weights for All Applicants in the Sample) 


Weight Variable N Sum Minimum Maximum 


Appl_wgt 2,074 2,074 LOO 1.00 

NOTE: Appl_wgt = the initial weight for all applicants in the sample (sums to the population total number of applicants) 


Exhibit B.9: Size of Target Population Within Each Weighting Class (Sum of Weights for All 
Applicants in the Sample) 


Weighting Class 

N 

Sum 

Minimum 

Maximum | 

ljPropensities 

-0-.35 

468 

468 

1.00 

1.00 

2)Propensities 

.35-. 68 

417 

417 

1.00 

1.00 

3)Propensities 

.68-. 84 

301 

301 

1.00 

1.00 

4)Propensities 

.84-. 88 

464 

464 

1.00 

1.00 

5)Propensities 

.88-. 91 

424 

424 

1.00 

1.00 


NOTE: Appl_wgt = initial weight for the applicant sample (sums to population total number of applicants within class) 


The sum of the initial weights of the 1,303 responding applicants is shown in Exhibit B.10. These are 
the "initial weights" because they are the sampling weights prior to adjustment for non-response. 
The weights of the 1,303 responding applicants sum to a number that is smaller than the size of the 
target population. Exhibit B.ll shows the weights summed within each of the five weighting classes. 
The sum shown for the y' th weighting class ( j = 1,...,5) corresponds to the term: 

X w <r 

i g responders 

For example, the initial applicant weights of the n responding applicants in the first weighting class 
sum to: 


2 >„= 37 

i g responders 


Exhibit B.10: Initial (Unadjusted) Weights of Responding Applicants (Sum of Initial Weights for 
All 1,303 Responding Applicants) 


I Variable N Sum 

Minimum 

Maximum 

Appl_wgt 1,303 1,303 

1.00 

1.00 

NOTE: Appl_wgt = the initial weight for the applicant sample 
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Exhibit B.ll: Initial (Unadjusted) Weights of Responding Applicants by Weighting Class (Sum of 
Initial Weights for All Responding Applicants) 


I Weighting Class 

N 

Sum 

Minimum 

Maximum j 

l)Propensities 

~.0-.35 

37 

37 

1.00 

1.00 

2)Propensities 

.35-. 68 

257 

257 

1.00 

1.00 

3)Propensities 

.68-. 84 

228 

228 

1.00 

1.00 

4)Propensities 

00 

00 

*3- 

00 

403 

403 

1.00 

1.00 

5)Propensities 

.88-. 91 

378 

378 

1.00 

1.00 


NOTE: Appl_wgt = initial weight for the applicant sample (sums to population total number of applicants within class) 


The inflation factors for each of the five weighting classes are shown in Exhibit B.12. The inflation 
factors correspond to the term: 


I >« 

k g responders & nonresponders 
i g responders 

For example, the inflation factor for the first weighting class is: 


468 


12.6486 


Exhibit B.12: Inflation Factors Within Weighting Classes 



Weighting Class 

All Applicants 

l)Propensities ~0-.35 

12.6486 

2)Propensities .35-. 68 

1.6226 

3)Propensities .68-. 84 

1.3202 

4)Propensities .84-.88 

1.1514 

5)Propensities .88-. 91 

1.1217 


Exhibits B.13 and B.14 show the sums of the non-response adjusted weights for the 1,303 
responding applicants, overall and by weighting class. The adjusted weights sum to the size of the 
target population. The numbers shown in Exhibits A.13 and A.14 correspond to the term: 

Z Z<* 

j i g responders 

For example, the non-response adjusted applicant weights of the 1,303 responder applicants sum to 

Z £>f= 2074 . 

j i g responders 

and, for example, the non-response adjusted applicant weights of the 378 responding applicants in 
the bottom weighting class sum to 



i g responders 


424 . 
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Exhibit B.13: Sum of Nonresponse Adjusted Weights for 1039 Responding Applicants 


Variable N Sum Minimum Maximum 


appl_wgt_adj 1303 2074 1.1216931 12.6486486 

NOTE: appl_wgt_adj = the adjusted weight for the applicant sample 


Exhibit B.14: Sum of Nonresponse Adjusted Weights for 1039 Responding Applicants by 
Weighting Class 


Weighting Class 

N 

Sum 

Minimum 

Maximum 1 

l)Propensities 

-0-.35 

37 

468 

12.6486 

12.6486 

2)Propensities 

.35-. 68 

257 

417 

1.6226 

1.6226 

3)Propensities 

.68-. 84 

228 

301 

1.3202 

1.3202 

4)Propensities 

.84-. 88 

403 

464 

1.1514 

1.1514 

5)Propensities 

.88-. 91 

378 

424 

1.1217 

1.1217 


NOTE: Appl_wft_adj = the adjusted weight for the applicant sample 


Item Non-Response 

Item non-response refers to one or more specific uncompleted items on an otherwise 
completed/returned questionnaire. Since the amount of missing data on an individual item was 
modest (<5% across all returned surveys), descriptive statistics were calculated on only the non- 
missing items, which is equivalent to an assumption that missing data on an item are missing 
completely at random. The amount of missing data for each item is presented in all tables/figures 
included in reports. 

Where necessary for the impact analyses, distinct approaches to imputing values were taken 
depending on whether data were missing for an item used to construct a covariate or predictor 
variable, or an outcome variable. For impact analyses where missing data on covariate or predictor 
variables require imputation to prevent having to omit those respondents from the analysis, a 
"dummy-variable" method was used. This method entailed (i) creating a dummy variable that equals 
"1" if the value of the variable is missing and "0" otherwise, (ii) adding the dummy variable to the 
impact model as a covariate, and (iii) replacing the missing value of the original variable with 
predicted values from a logistic or linear regression model (see appendix C for more details). 

If the missing data occurred in an item used to construct an outcome— that is, one of the primary 
outcomes of interest (for example, the post-fellowship number of publications produced with a 
foreign co-author)— no imputation was conducted. 
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Appendix C: Detailed Description of Impact Analysis 


This appendix describes the methodology used to estimate the effects of the EAPSI award on its 
recipients. First, key features of the quasi-experimental design are reviewed in order to provide 
context for a general discussion of the use of propensity score analysis (PSA) in mitigating selection 
bias. Next, the steps used to match awardees and non-awardees are presented in detail, including 
the estimated logistic model and the resulting distribution of awardees and non-awardees in the 
propensity strata. Finally, specifications of the impact models and sensitivity analyses conducted are 
presented. 


Propensity Score Matching 


One of the main purposes of this evaluation was to estimate the effect of EAPSI on its participants. If 
a program brings about changes in its participants, then these individuals should have different 
outcomes, post-participation, than they would have had in the absence of program participation. 

Questions about the impact of a program seek to determine whether any observed differences 
between participants and non-participants can be attributed to the program itself rather than to 
other, non-program related causes. One potential source of other causes for differences between 
the two groups is pre-existing characteristics that could affect both selection into the program and 
post-program differences, often called "selection bias". Propensity score matching (PSM) was used 
to mitigate selection bias for this evaluation. 

Propensity score matching is one type of propensity score analysis (PSA) wherein participants in a 
program are matched to non-participants on the basis of their "participation propensity score." This 
technique uses pre-treatment characteristics to determine the probability (i.e., the propensity 
score) that applicants would be selected for the treatment (namely, an EAPSI award) based on 
known pre-existing characteristics). After assigning a propensity score to each individual, applicants 
are placed into blocks (or matching strata) such that the actual EAPSI awardees and the unfunded 
applicants within each block have approximately equal predicted propensity to be in the treated 
group. The quasi-experimental estimates of the impact of the program can then be obtained by 
comparing the outcomes of awardees and non-awardees within each propensity block and 
aggregating the differences across the blocks. This is accomplished by including terms for the 
propensity blocks in the models used for analysis. 

PSM was performed via the following four steps: 

Step 1 : Identify the pre-treatment characteristics that will be used in the propensity score 
model to match fellows and unfunded applicants. 

The pre-treatment characteristics to be used in the propensity score model to match fellows and 
unfunded applicants were identified. These characteristics included variables that both predicted 
receiving the fellowship and that might also affect the outcomes of interest. They were taken from 
NSF extant data and applicant survey data. Exhibit C.l shows the variables used in the propensity 
score model. 
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Some of the characteristics to be used in the PSA had missing data. If these missing data had been 
ignored, records that had missing data in the PSA matching would have been lost. A recommended 
approach for addressing this issue is to do a simple single imputation of the missing covariates and 
include missing data indicators in the propensity score model. This method essentially matches both 
on the observed values and on the missing data patterns (Stuart 2010). 4 Although it cannot balance 
the missing data values themselves this method will yield balance on the covariates and the missing 
data patterns (Rosenbaum and Rubin, 1984). 5 

Single imputation substitutes a missing value with a definite value following an established 
procedure. Predicted values from a logistic or linear regression model were used to impute a 
definite value for missing values in this study. The outcomes for these models were the PSA 
variables that had missing data and the predictors were all PSA variables with non-missing data. 
Imputation was done first for the variable with the least amount of missing. This variable was then 
added to the right hand side of the model for the next variable with the least amount of missing. 

This process continued through to the variable with the most number of cases missing with all of the 
other PSA variables. For binary variables a imputation was done using randomly generated a 0 or 1 
using a binomial distribution with p=predicted probability from the model discussed above. For 
continuous variable the predicted values generated from the model was used as the imputed value. 
Imputation flags were created to indicate if an observation was imputed or not to include in the PSA 
model. Exhibit C.2 shows the distribution of each variable overall and by award status prior and 
post-imputation. 


Stuart, E.A. 2010. Matching methods for casual inference: A review and a look forward. Statistical Science, 
25(1), 1-21. 

Rosenbaum, P.R. and Rubin, D.B. 1984. Reducing bias in observational studies using subclassification on 
the propensity score. Journal of the American Statistical Association 79, 516 -524. 
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Exhibit C.l: Pre-Award Data Used to Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact Models 


Pre-Award Characteristic 

Reason for Inclusion in Propensity 
Score Model 1 

Data Source(s) 

Type 

Definition 

Mean proposal score 

Proposal score indicates quality of 
application 

NSF Extant Data 

Continuous 

Average score across reviews (1-5) 

Cohort year 

Control for cohort differences 

NSF Extant Data 

Dichotomous 

2002 (0/1) 

2003 (0/1) 

2004 (0/1) 

2005 (0/1) 

2006 (0/1) 

2007 (0/1) 

2008 (0/1) 
2009 (0/1) 


Gender 

Preference given for females (women 
encouraged to apply) 3 

EAPSI Applicant 
Survey: Gl, NSF Extant 
Data 

Dichotomous 

l=Female 

0=Male 

Under-represented minority 
status 

Preference given for under- 
represented minority status (members 
of these groups were encouraged to 
apply) 3 

EAPSI Applicant 
Survey: G2, G3, NSF 
Extant Data 

Dichotomous 

l=Other race(s)/ethnicity 
0=Asian Only or White Only 

Citizenship status 

US citizenship (birth, naturalized) or 
permanent residency required 3 

EAPSI Applicant 
Survey: G4, G4a 

Dichotomous 

1=US Citizen/Permanent Resident 
0=Non-US Citizen or Permanent 
Resident 

Disability status 

Preference given to disabled applicant 
(disabled applicants encouraged to 
apply) 3 

EAPSI Applicant 
Survey: G5, NSF Extant 
Data 

Dichotomous 

l=Disabled 

0=Not-Disabled 
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Exhibit C.l: Pre-Award Data Used to Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact Models 


Pre-Award Characteristic 

Reason for Inclusion in Propensity 
Score Model 1 

Data Source(s) 

Type 

Definition 

STEM discipline 

Desire for disciplinary balance in 
program portfolio a 

EAPSI Applicant 
Survey: A7 

Dichotomous 

Biological, agricultural, or 
environmental life sciences (1/0) 


Physical and related sciences 
(includes Chemistry except 
biochem; earth, atmospheric, 
ocean sciences; physics) (1/0) 
Computer and information sciences 
& Mathematics and statistics (1/0) 
Psychology & social sciences 
&related science (1/0) 

Engineering (1/0) 

Health (non-STEM) 1/0) 


Undergraduate GPA 

EAPSI application requests 
undergraduate transcript 3 

EAPSI Applicant 
Survey: Cl 

Continuous 

l=Less than 1.25 (Mostly D's or 
below) 

2=1.25 - 1.74 GPA (About half 
C' s/half D's) 

3=1.75 - 2.24 GPA (Mostly C's) 
4=2.25 - 2.74 GPA (About half 
B' s/half C's) 

5=2.75 - 3.24 GPA (Mostly B's) 
6=3.25 - 3.74 GPA (About half 
A's/half B's) 

7=3.75 - 4.00 GPA (Mostly A's) 

Graduate degree program 

Master's- or Doctoral-level (M.D. 

EAPSI Applicant 

Dichotomous 

l=Doctorate 


included) program 3 

Survey: A5 


0=Master's 

Study-abroad as an 

Prior international experience 

EAPSI Applicant 

Dichotomous 

l=Prior experience studying abroad 

undergraduate or graduate 

favorable for EAPSI applicant 3 

Survey:C2a, C2b 


0=No Prior experience studying 





abroad 
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Exhibit C.l: Pre-Award Data Used to Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact Models 



Reason for Inclusion in Propensity 




Pre-Award Characteristic 

Score Model 1 

Data Source(s) 

Type 

Definition 


Prior visit to host location 

Prior exposure to host location 
favorable for applicant (shows 
evidence of "willingness to live in and 
adapt to foreign cultures") a 

EAPSI Applicant 
Survey:C3, C6 

Continuous 

0=Did not live or study abroad 
l=Lived in a foreign country but not 
the country applied OR Lived in a 
foreign country for 6 months or 
longer OR Attended school in 
another country outside the US 
2= Lived in country applied to 

Prior relevant language(s) 

Language aptitude or achievement not 
required but likely favorable for 
applicant 3 

EAPSI Applicant 
Survey:C4 

Dichotomous 

1= Prior relevant language(s) 
0=No prior relevant language(s) 

Participation in international 
club(s) 

Favorable for EAPSI applicant 3 

EAPSI Applicant 
Survey:C5 

Dichotomous 

1= Participated in international 
club(s) 

0=Did not participate in 
international club(s) 

Prior international 
collaboration 

Beneficial to applicant 3 

EAPSI Applicant 
Survey: C6, C7b 

Dichotomous 

l=Prior international collaboration 
0= No prior international 
collaboration 

Letter of support from host 

Strongly favorable for EAPSI applicant 3 

EAPSI Applicant 
Survey:C7a 

Dichotomous 

l=Yes letter of support from host 
0=No letter of support from host/I 
do not recall 

Prior international exposure 

Prior exposure to foreign colleagues or 
former program fellow favorable to 
applicant 

EAPSI Applicant 
Survey:C6 

Dichotomous 

l=Prior exposure to foreign 
colleagues or former program 
fellow 

0=No prior exposure to foreign 
colleagues or former program 
fellow 

Link between US, host 
institutions 

Likely to be beneficial to applicant 

EAPSI Applicant 
Survey:C7c 

Dichotomous 

l=Link between US, host 
institutions 

0=No Link between US, host 
institutions 

Total pre-award publications 

Prior record of achievement favorable 3 

EAPSI Applicant 
Survey:C8 

Continuous 

Total pre-award publication 
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Exhibit C.l: Pre-Award Data Used to Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact Models 


Pre-Award Characteristic 

Reason for Inclusion in Propensity 
Score Model 1 

Data Source(s) 

Type 

Definition 

% publications w/foreign 
collaborator 

Likely to be beneficial to applicant 

EAPSI Applicant 
Survey:C8 

Continuous 

Percent of publications with foreign 
collaborator 

National post-collegiate 
fellowship 

Prior record of achievement favorable 3 

EAPSI Applicant 
Survey:C9 

Dichotomous 

l=Received a national post- 
collegiate fellowship 
0=Did not received any national 
post-collegiate fellowship 

3 These prerequisites are taken from NSF solicitations 99-152, 02-007, 02-174, 03-608, 05-617, 06-602, 07-584, 08-603, and 10-591. 
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Exhibit C.2: Distribution of Pre-Award Data Used To Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact 

Models 















Prior to Imputation 


Post-Imputation 

Pre-Award 



N 



Mean/ 


N 



Mean/ 

Characteristic 

Variable 

N 

miss 

Min 

Max 

Percent 

N 

miss 

Min 

Max 

Percent 

Among All Applicants 

Mean Proposal 
Score 

finalscore 

1303 

0 

1 

5 

3.86 

1303 

0 

1 

5 

3.86 

Cohort Year 

dum2000 

1303 

0 

0 

1 

4.7% 

1303 

0 

0 

1 

4.7% 


dum2001 

1303 

0 

0 

1 

3.4% 

1303 

0 

0 

1 

3.4% 


dum2002 

1303 

0 

0 

1 

6.8% 

1303 

0 

0 

1 

6.8% 


dum2003 

1303 

0 

0 

1 

4.7% 

1303 

0 

0 

1 

4.7% 


dum2004 

1303 

0 

0 

1 

8.7% 

1303 

0 

0 

1 

8.7% 


dum2005 

1303 

0 

0 

1 

10.4% 

1303 

0 

0 

1 

10.4% 


dum2006 

1303 

0 

0 

1 

10.1% 

1303 

0 

0 

1 

10.1% 


dum2007 

1303 

0 

0 

1 

12.3% 

1303 

0 

0 

1 

12.3% 


dum2008 

1303 

0 

0 

1 

22.9% 

1303 

0 

0 

1 

22.9% 

Gender 

cr nrbias gender 

1303 

0 

0 

1 

43.7% 

1303 

0 

0 

1 

43.7% 

Under-Represented 
Minority Status 

cr_nrbias_URM 

1272 

31 

0 

1 

12.7% 

1303 

0 

0 

1 

13.0% 

Citizenship Status 

desc_ctzn 

1269 

34 

0 

1 

97.9% 

1303 

0 

0 

1 

97.9% 

Disability status 

cr_nrbias_hdcap 

1269 

34 

0 

1 

0.6% 

1303 

0 

0 

1 

0.8% 

STEM Discipline 

dumSTEMBiology 

1303 

0 

0 

1 

29.0% 

1303 

0 

0 

1 

29.0% 


dumSTEMComputer 

1303 

0 

0 

1 

8.6% 

1303 

0 

0 

1 

8.6% 


dumSTEMEngineer 

1303 

0 

0 

1 

21.6% 

1303 

0 

0 

1 

21.6% 


dumSTEMHealth 

1303 

0 

0 

1 

9.3% 

1303 

0 

0 

1 

9.3% 


dumSTEMPhysical 

1303 

0 

0 

1 

27.5% 

1303 

0 

0 

1 

27.5% 


dumSTEMSocial 

1303 

0 

0 

1 

4.1% 

1303 

0 

0 

1 

4.1% 

Undergraduate 

GPA 

UGGPA 

1289 

14 

2 

7 

6.24 

1303 

0 

2 

7 

6.23 

Graduate Degree 
Program 

PhDYes 

1301 

2 

0 

1 

78.2% 

1303 

0 

0 

1 

78.2% 

Study Abroad as an 
undergraduate or 
graduate 

StudAbr 

1294 

9 

0 

1 

37.9% 

1303 

0 

0 

1 

37.8% 
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Exhibit C.2: Distribution of Pre-Award Data Used To Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact 
Models 





Prior to Imputation 


Post-Imputation 

Pre-Award 



N 



Mean/ 


N 



Mean/ 

Characteristic 

Variable 

N 

miss 

Min 

Max 

Percent 

N 

miss 

Min 

Max 

Percent 

Prior visit to host 
location 

PriorSiteVis 

1300 

3 

0 

2 

0.65 

1303 

0 

0 

2 

0.65 

Prior relevant 
language 

PriorLang 

1301 

2 

0 

1 

23.6% 

1303 

0 

0 

1 

23.6% 

Participation in 
international clubs 

IntIClub 

1295 

8 

0 

1 

42.1% 

1303 

0 

0 

1 

42.0% 

Prior international 
collaboration 

PriorCollab 

1292 

11 

0 

1 

37.7% 

1303 

0 

0 

1 

37.7% 

Letter of support 
from host 

LetterSupp 

1296 

7 

0 

1 

70.3% 

1303 

0 

0 

1 

70.3% 

Prior international 
exposure 

IntlExpose 

1300 

3 

0 

1 

31.9% 

1303 

0 

0 

1 

31.8% 

Link between host, 
US institution 

LinkHost 

1299 

4 

0 

1 

36.4% 

1303 

0 

0 

1 

36.3% 

Total Pre-Award 
publications 

AllWorks 

1130 

184 

0 

75 

4.04 

1303 

0 

0 

75 

3.91 

Percent of 
publications with a 
foreign 
collaborator 

PercentWorks 

1115 

188 

0 

100 

9.02 

1303 

0 

0 

100 

9.00 

National post- 

collegiate 

fellowship 

fellowship 

1286 

17 

0 

1 

22.0% 

1303 

0 

0 

1 

22.1% 

Among Awardees Only 

Mean Proposal 
Score 

finalscore 

938 

0 

1.3 

5 

4.14 

938 

0 

1.3 

5 

4.14 

Cohort Year 

dum2000 

938 

0 

0 

1 

6.5% 

938 

0 

0 

1 

6.5% 


dum2001 

938 

0 

0 

1 

4.7% 

938 

0 

0 

1 

4.7% 


dum2002 

938 

0 

0 

1 

7.4% 

938 

0 

0 

1 

7.4% 


dum2003 

938 

0 

0 

1 

6.2% 

938 

0 

0 

1 

6.2% 


dum2004 

938 

0 

0 

1 

8.6% 

938 

0 

0 

1 

8.6% 
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Exhibit C.2: Distribution of Pre-Award Data Used To Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact 
Models 





Prior to Imputation 


Post-Imputation 

Pre-Award 



N 



Mean/ 


N 



Mean/ 

Characteristic 

Variable 

N 

miss 

Min 

Max 

Percent 

N 

miss 

Min 

Max 

Percent 


dum2005 

938 

0 

0 

1 

11.3% 

938 

0 

0 

1 

11.3% 


dum2006 

938 

0 

0 

1 

10.8% 

938 

0 

0 

1 

10.8% 


dum2007 

938 

0 

0 

1 

12.0% 

938 

0 

0 

1 

12.0% 


dum2008 

938 

0 

0 

1 

15.8% 

938 

0 

0 

1 

15.8% 


dum2009 

938 

0 

0 

1 

16.7% 

938 

0 

0 

1 

16.7% 

Gender 

cr_nrbias_gender 

938 

0 

0 

1 

42.0% 

938 

0 

0 

1 

42.0% 

Under-Represented 
Minority Status 

cr_nrbias_URM 

919 

19 

0 

1 

11.5% 

938 

0 

0 

1 

12.0% 

Citizenship Status 

desc_ctzn 

915 

23 

0 

1 

98.4% 

938 

0 

0 

1 

98.3% 

Disability status 

cr_nrbias hdcap 

912 

26 

0 

1 

0.7% 

938 

0 

0 

1 

0.9% 

STEM Discipline 

dumSTEMBiology 

938 

0 

0 

1 

28.3% 

938 

0 

0 

1 

28.3% 


dumSTEMComputer 

938 

0 

0 

1 

8.7% 

938 

0 

0 

1 

8.7% 


dumSTEMEngineer 

938 

0 

0 

1 

21.4% 

938 

0 

0 

1 

21.4% 


dumSTEMHealth 

938 

0 

0 

1 

8.8% 

938 

0 

0 

1 

8.8% 


dumSTEMPhysical 

938 

0 

0 

1 

28.7% 

938 

0 

0 

1 

28.7% 


dumSTEMSocial 

938 

0 

0 

1 

4.1% 

938 

0 

0 

1 

4.1% 

Undergraduate 

GPA 

UGGPA 

928 

10 

3 

7 

6.27 

938 

0 

3 

7 

6.27 

Graduate Degree 
Program 

PhDYes 

938 

0 

0 

1 

80.9% 

938 

0 

0 

1 

80.9% 

Study Abroad as an 
undergraduate or 
graduate 

StudAbr 

935 

3 

0 

1 

37.9% 

938 

0 

0 

1 

37.7% 

Prior visit to host 
location 

PriorSiteVis 

937 

1 

0 

2 

0.65 

938 

0 

0 

2 

0.65 

Prior relevant 
language 

PriorLang 

937 

1 

0 

1 

27.3% 

938 

0 

0 

1 

27.3% 

Participation in 
international clubs 

IntIClub 

936 

2 

0 

1 

40.2% 

938 

0 

0 

1 

40.1% 

Prior international 
collaboration 

PriorCollab 

933 

5 

0 

1 

37.4% 

938 

0 

0 

1 

37.3% 
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Exhibit C.2: Distribution of Pre-Award Data Used To Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact 

Models 















Prior to Imputation 




Post-Imputation 


Pre- Award 



N 



Mean/ 


N 



Mean/ 

Characteristic 

Variable 

N 

miss 

Min 

Max 

Percent 

N 

miss 

Min 

Max 

Percent 

Letter of support 
from host 

LetterSupp 

935 

3 

0 

1 

73.0% 

938 

0 

0 

1 

73.1% 

Prior international 
exposure 

IntlExpose 

937 

1 

0 

1 

33.8% 

938 

0 

0 

1 

33.8% 

Link between host, 
US institution 

LinkHost 

936 

2 

0 

1 

40.1% 

938 

0 

0 

1 

40.0% 

Total Pre-Award 
publications 

AllWorks 

815 

123 

0 

75 

4.08 

938 

0 

0 

75 

3.97 

Percent of 
publications with a 
foreign 
collaborator 

PercentWorks 

812 

126 

0 

100 

9.59 

938 

0 

0 

100 

9.48 

National post- 

collegiate 

fellowship 

fellowship 

929 

9 

0 

1 

23.6% 

938 

0 

0 

1 

23.5% 

Among Declines Only 

Mean Proposal 
Score 

finalscore 

365 

0 

1 

5 

3.13 

365 

0 

1 

5 

3.13 

Cohort Year 

dum2000 

365 

0 

0 

0 

0.0% 

365 

0 

0 

0 

0.0% 


dum2001 

365 

0 

0 

0 

0.0% 

365 

0 

0 

0 

0.0% 


dum2002 

365 

0 

0 

1 

5.5% 

365 

0 

0 

1 

5.5% 


dum2003 

365 

0 

0 

1 

0.8% 

365 

0 

0 

1 

0.8% 


dum2004 

365 

0 

0 

1 

9.0% 

365 

0 

0 

1 

9.0% 


dum2005 

365 

0 

0 

1 

8.2% 

365 

0 

0 

1 

8.2% 


dum2006 

365 

0 

0 

1 

8.2% 

365 

0 

0 

1 

8.2% 


dum2007 

365 

0 

0 

1 

12.9% 

365 

0 

0 

1 

12.9% 


dum2008 

365 

0 

0 

1 

41.4% 

365 

0 

0 

1 

41.4% 


dum2009 

365 

0 

0 

1 

14.0% 

365 

0 

0 

1 

14.0% 

Gender 

cr_nrbias_gender 

365 

0 

0 

1 

47.9% 

365 

0 

0 

1 

47.9% 

Under-Represented 
Minority Status 

cr_nrbias_URM 

353 

12 

0 

1 

15.6% 

365 

0 

0 

1 

15.6% 
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Exhibit C.2: Distribution of Pre-Award Data Used To Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact 
Models 





P 

rior to Imputation 


Post-Imputation 

Pre-Award 



N 




Mean/ 


N 



Mean/ 

Characteristic 

Variable 

N 

miss 


Min 

Max 

Percent 

N 

miss 

Min 

Max 

Percent 

Citizenship Status 

desc_ctzn 

354 

ii 


0 

1 

96.6% 

365 

0 

0 

1 

96.7% 

Disability status 

cr_nrbias_hdcap 

357 

8 

0 

1 

0.6% 

365 

0 

0 

1 

0.5% 

STEM Discipline 

dumSTEMBiology 

365 

0 

0 

1 

31.0% 

365 

0 

0 

1 

31.0% 


dumSTEMComputer 

365 

0 

0 

1 

8.2% 

365 

0 

0 

1 

8.2% 


dumSTEMEngineer 

365 

0 

0 

1 

21.9% 

365 

0 

0 

1 

21.9% 


dumSTEMHealth 

365 

0 

0 

1 

10.4% 

365 

0 

0 

1 

10.4% 


dumSTEMPhysical 

365 

0 

0 

1 

24.4% 

365 

0 

0 

1 

24.4% 


dumSTEMSocial 

365 

0 

0 

1 

4.1% 

365 

0 

0 

1 

4.1% 

Undergraduate 

GPA 

UGGPA 

361 

4 

2 

7 

6.14 

365 

0 

2 

7 

6.14 

Graduate Degree 
Program 

PhDYes 

363 

2 

0 

1 

71.1% 

365 

0 

0 

1 

71.2% 

Study Abroad as an 
undergraduate or 
graduate 

StudAbr 

359 

6 

0 

1 

37.9% 

365 

0 

0 

1 

37.8% 

Prior visit to host 
location 

PriorSiteVis 

363 

2 

0 

2 

0.64 

365 

0 

0 

2 

0.64 

Prior relevant 
language 

PriorLang 

364 

1 

0 

1 

14.0% 

365 

0 

0 

1 

14.0% 

Participation in 
international clubs 

IntIClub 

359 

6 

0 

1 

47.1% 

365 

0 

0 

1 

46.8% 

Prior international 
collaboration 

PriorCollab 

359 

6 

0 

1 

38.4% 

365 

0 

0 

1 

38.6% 

Letter of support 
from host 

LetterSupp 

361 

4 

0 

1 

63.2% 

365 

0 

0 

1 

63.0% 

Prior international 
exposure 

IntlExpose 

363 

2 

0 

1 

27.0% 

365 

0 

0 

1 

26.8% 

Link between host, 
US institution 

LinkHost 

363 

2 

0 

1 

27.0% 

365 

0 

0 

1 

26.8% 

Total Pre-Award 

AllWorks 

304 

61 

0 

33 

3.91 

365 

0 

0 

33 

3.76 


publications 
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Exhibit C.2: Distribution of Pre-Award Data Used To Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact 
Models 





Prior to Imputation 




Post-Imputation 


Pre-Award 



N 



Mean/ 


N 



Mean/ 

Characteristic 

Variable 

N 

miss 

Min 

Max 

Percent 

N 

miss 

Min 

Max 

Percent 

Percent of 
publications with a 
foreign 
collaborator 

PercentWorks 

303 

62 

0 

100 

7.49 

365 

0 

0 

100 

7.80 

National post- 

collegiate 

fellowship 

fellowship 

357 

8 

0 

1 

17.9% 

365 

0 

0 

1 

18.6% 
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Step 2: Fit a logistic model that predicts the probability of being awarded a fellowship based 
on pre-treatment characteristics. 

The participation propensity score for each individual was estimated using a logistic model with the 
identified pre-treatment characteristics (see Exhibit C.3) as the independent variables and receipt of 
the fellowship (dummy-coded as 0 or 1) as the dependent variable. In general, variables were not 
excluded from the logistic model merely because of a lack of significance, i.e., variables were 
included regardless of whether they predicted treatment. Collinearity was accepted among the 
predictors because the model was not intended to predict anything outside the sample space. 
Exhibit C.3 displays the resulting logistic model. The coefficients from this model were used to 
estimate the propensity score for each individual, which represents the probability of receiving a 
fellowship. 


Exhibit C.3: Propensity Score Logistic Model 


Variable 

Coefficient 

Standard Error 

Prob > | z | 

finalscore 

2.79 

0.18 

0.00 

dum2002 

0.47 

0.43 

0.28 

dum2003 

-1.17 

0.77 

0.13 

dum2004 

0.71 

0.40 

0.07 

dum2005 

0.33 

0.39 

0.40 

dum2006 

2.01 

0.51 

0.00 

dum2007 

1.28 

0.32 

0.00 

dum2008 

2.90 

0.29 

0.00 

cr_nrbias_gender 

0.22 

0.20 

0.29 

cr_nrbias_urm 

-0.34 

0.27 

0.22 

Cr_nrbias_urmjmp_flg 

-0.24 

0.66 

0.71 

cr_nrbias_hdcap 

0.16 

1.10 

0.89 

Cr_nrbias_hdcapjmp_flg 

-0.02 

0.74 

0.98 

dumstemcomputer 

-1.52 

0.41 

0.00 

dumstemengineer 

0.05 

0.26 

0.85 

dumstemhealth 

0.29 

0.38 

0.45 

dumstemphysical 

-0.59 

0.30 

0.05 

dumstemsocial 

-0.86 

0.52 

0.10 

Uggpa 

0.15 

0.12 

0.21 

uggpajmpjlg 

-3.30 

1.61 

0.04 

Phdyes 

-0.04 

0.28 

0.88 

Studabr 

0.18 

0.24 

0.45 

studabr_imp_flg 

1.99 

1.19 

0.10 

Priorsitevis 

-0.18 

0.16 

0.25 

Priorlang 

-1.67 

0.29 

0.00 

Intlclub 

0.29 

0.20 

0.14 

intlclubjmp_flg 

2.43 

2.40 

0.31 

priorcollab 

1.33 

0.43 

0.00 

PriorcollabJmp_flg 

1.88 

1.43 

0.19 

lettersupp 

-0.11 

0.22 

0.61 

Lettersupp_imp_flg 

-1.04 

4.52 

0.82 

intlexpose 

-0.25 

0.21 

0.24 

linkhost 

-0.95 

0.21 

0.00 

linkhostJmp_flg 

-1.50 

4.69 

0.75 

allworks 

0.01 

0.02 

0.64 

allworks_imp_flg 

0.38 

1.47 

0.80 
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Exhibit C.3: Propensity Score Logistic Model 


3 Variable 

Coefficient 

Standard Error 

Prob > | z | I 

percentworks 

0.01 

0.02 

0.47 

PercentworksJmp_flg 

-0.29 

1.46 

0.84 

fellowship 

0.14 

0.24 

0.57 

FellowshipJmp_flg 

0.80 

1.11 

0.47 

phdyes_prciocollab 

-0.92 

0.45 

0.04 

studabr_stemphys 

0.25 

0.46 

0.58 

dum2006_priorcollab 

-1.37 

0.63 

0.03 

percentworkssquare 

0.00 

0.00 

0.83 

gender_dum2006 

-1.00 

0.63 

0.11 

_cons 

-11.34 

6.00 

0.06 


Number of observations =1303 Pseudo R 2 = 0.4860 

LR chi 2 (42) = 751.12 Log-likelihood = -397.20877 

Prob > chi 2 = 0.0000 


NOTES: In estimating the propensity score through a probability model, the choice of which interaction term to include is 
determined solely by the need to condition fully on the observable characteristics that make up the assignment 
mechanism. When covariates are not balanced within a particular stratum, the solution adopted is to divide the stratum 
into finer strata and test again for no difference in the distribution of covariates within the finer strata. If however, some 
covariates still remain unbalanced, the score may be poorly estimated which suggests that additional terms (interaction 
or higher-order terms) of the unbalanced covariates should be added to the logistic specification to control better for 
these characteristics. This procedure is repeated until the covariates are balanced. See Deheija & Wahba (2002). 
Propensity score-matching methods for non-experimental casual studies. The Review of Economics and Statistics, 84(1): 
pg 161. * 6 

Step 3: Use the estimated propensity scores to create matched sets of fellows and unfunded 
applicants. 

The estimated propensity scores were used to create matched sets of fellows and unfunded 
applicants. Propensity scores can be utilized in a number of ways, including matching, stratification, 
weighting, and regression adjustment. 7 Stratification (also called interval matching) was initially 
identified as the primary method of matching; after the sensitivity checks were conducted the 3:1 
matching method (see the sensitivity section below for discussion) was selected as the primary 
method for matching. Once the propensity scores were assigned, the region of common support 
was defined as [.02034455, .99989093]. Individuals were identified and excluded from the analyses 
if they were outside of the "common support" group - the range of common scores across fellows 
and unfunded applicants. Enforcing the common support is important to ensure the similarity of the 
matched non-awardees to awardees. 8 0 awardees and 38 non-awardees were dropped because 
their propensity score was outside of this range. 


See Deheija and Wahba. 2002. Propensity score-matching methods for non-experimental casual studies. 

The Review of Economics and Statistics, 84(1), 161. 

7 Keisuke, H., Imbens, G.W., and Ridder, G. 2003. Efficient estimation of average treatment effects Using 
the estimated propensity score. Econometrica, 71(4), 1161-89; Morgan, S.L. and Harding, D. J. 2006. 
Matching estimators of causal effects: prospects and pitfalls in theory and practice. Sociological Methods 
and Research, 35(1), 3-60; and Abadie, A., and Imbens, G. W. 2009. Matching on the Estimated Propensity 
Score. NBER Working Paper. 

8 Rosenbaum and Rubin, 1984; Caliendo, M. and Kopeinig, S. 2007. Some practical guidance for the 
implementation of propensity score matching. Journal of Economic Surveys, 22(1), 31-72. 
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The remaining individuals were divided into strata within which the awardee and non-awardee 
groups had statistically the same 9 mean propensity score. Within each of these strata tests were 
conducted to identify significant differences between the treatment and comparison group on the 
independent variables. The model was adjusted until all such differences were removed 10 . 

The treatment and comparison groups were divided into 6 strata based on their propensity scores. 
Research has indicated that at least five strata are generally sufficient for removing 90 percent or 
more of the bias due to the covariates. * 11 Exhibit C.4 shows the distribution of the treatment and 
comparison group members by propensity score strata. Exhibit C.5 shows a histogram of propensity 
scores for the treatment and comparison groups. 


Exhibit C.4: Number of Awardee and Non-Awardee in each Propensity Score Block 


| Inferior of Block of Pscore 

Non-Awardee 

Awardee 

Total 

.0203445 

118 

9 

127 

.2 

74 

27 

101 

.4 

54 

60 

114 

.6 

45 

111 

156 

.8 

18 

120 

138 

.9 

18 

611 

629 

Total 

327 

938 

1265 

Number outside the region of common support 

38 

0 

38 


Based on level p<0.01 t-tests. 

10 The Intercooled STATA10 pscore process was used to obtain propensity score balance. 

11 Rosenbaum and Rubin, 1984. 
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Exhibit C.5: Histograms of Propensity Scores 



Awardees 


Step 4: Test whether there are any differences between the awardees and non-awardees 
within each propensity score strata. 

This analysis was conducted in two ways. First, t-tests were used to determine if there were any 
difference for awardees and non-awardees within each stratum for each pre-treatment 
characteristic . 12 As this test is sensitive to sample size (i.e., they tend fail to detect sizable 
differences in small samples, but detect slight differences in larger samples), this analysis was 
supplemented using standardized differences . 13 The standardized difference of a matching 
characteristic between awardees and non-awardees in a given propensity score stratum is 
calculated using the following formula: 


Dehejia and Wahba, 2002; Agodini, R., and Dynarski, M. 2004. Are experiments the only option? A look at 
dropout prevention programs. Review of Economics and Statistics, 86(1), 180-94. 

Morgan, S.L., and Winship, C. 2008. Counterfactuals and causal inference: Methods and principles for 
social Research. New York: Cambridge University Press. 
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B x,s = 


I X T S X c,s I 


I — cr x,t H — <J"x,c 

2 2 


Where: 

X denotes the variable of interest; 

S denotes the stratum; 

T denotes the treatment group, and Cdenotes the comparison group; 

X T s and X c s denote the treatment and comparison group mean of X in stratum S ; and 

cr 2 x,r and a 2 x,c denote the overall variance of X in the treatment and comparison group, 

respectively. 

For regression adjustments to be trustworthy standardized differences of means should be less that 
0.25 (Rubin, 2001) 14 and you want very few of significant t-tests (Stuart 2010). 15 Checks were 
conducted for statistical balance and modified the models in step 2 by including interactions and 
higher-order terms of the unbalanced characteristics until satisfactory statistical balance was 
achieved i.e. models were improved until it was no longer possible to reduce the number of 
significant t-tests or standardized differences <0.25. For this study total of five models that achieved 
propensity score balance to achieve this result were fit. 

Pre-treatment characteristics and standardized differences between awardees and non-awardees 
within each of the 6 strata were examined. The sizes of the treatment-control differences expressed 
in standard deviation units (standardized differences) ranged from .010 to .910, with the largest 
differences in the block with the smallest awardee group size (block 1). Controls were included the 
impact model for any characteristics in the impact models whose standardized differences were 
greater than 0.25. 


Impact Models 


Following the propensity score matching, the impact of the EAPSI program was estimated by 
comparing fellows' outcomes to those of their comparison group to determine what fellows' 
expected outcomes would have been had they not received an EAPSI award. 

Estimation of Impacts 

After creating the propensity score strata, a multivariable regression model was used to estimate 
the impact of the program of interest. This regression model employed a number of matching 

14 Rubin, D.B. 2001. Using propensity scores to help design observational studies: Application to the tobacco 
litigation. Health Services and Outcomes Research Methodology 2, 169-188. 

15 Stuart, 2010. 
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characteristics and other variables (see Exhibit C.6) that are hypothesized to affect the outcomes of 
interest such as covariates. The inclusion of the matching characteristics in this model give us the 
chance to get a "doubly-robust" impact estimate since they would have been used twice: both in the 
propensity score model and in the estimation of impacts. 16 The following regression model was used 
to estimate the program impact: 17 


Impact Model 

[l] Y t =p„ + pj, + 2X7A/ +X/ 3 „, <M) X’ +«, 


b - 1 


b-\ 


j= 1 


7=1 


Where: 


Y t is the outcome of interest for individual i, 

T f is the treatment indicator for individual i (l=awardee , O=non-awardee), 

Sf is the indicator (dummy) variable for the j ,h propensity score stratum. The prototypical 
model includes the total number of strata ( b ) minus one strata indicators (j=l,2 ,..., b-1). The last 
stratum is the reference stratum and a dummy for this stratum is not included in the model, 

X" is the n ,h (n=l,2,...,N) covariate for individual i (such as gender, URM, etc.) that are grand- 
mean centered, and 

£, is the usual error term for individual Interpretation of Parameters 

Interpretation of the coefficients in the model is as follows: 

/ 3 0 is the covariate-adjusted mean value of the outcome for the non-awardees in the reference 
propensity score stratum, 

A) + P, +i (j=l,2,...,b-l) is the covariate-adjusted mean value of the outcome for the non- 
awardees in the j lh stratum, 

P j is the impact estimate for the reference stratum (i.e. the difference between the mean value 
of the outcome of the awardees and non-awardees in the reference stratum), 

P j+b 0=1,2,..., b-1) is the difference between the impact estimate for the j th stratum and the 
impact estimate for the reference stratum, 


Ho D.E., Imai K., King G., and Stuart E. A. 2007. Matching as nonparametric preprocessing for reducing 
model dependence in parametric causal inference. Political Analysis, 15, 199-236.; Morgan and Harding, 
2006. 

For illustrative purposes, the impact model is presented for continuous outcomes. For binary outcomes, a 
logistic model was fit structured similarly to the model in Equation 1. 
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Pi + Pj+b (j=l,2,...,b-l) is the impact estimate (i.e., the covariate adjusted difference between 
the outcomes of the awardees and non-awardees) for the j ,h stratum, and 

P n+ (b+ 3 ) (n=l,2,...,N) is the estimated overall relationship between the n' h covariate and the 
outcome controlling for other covariates. 

Treatment Effects 

Overall treatment effect (estimated impact) 

As seen, the model in Equation 1 allows for the estimation of separate treatment effect estimates 
for each propensity score stratum. More specifically, p { + p j+b (j=l,2,...,b-l) is the impact estimate 

for the j ,h ( j = 1, 2,..., b-1) stratum. In order to calculate an overall treatment effect estimate, the 
stratum-specific estimates are aggregated as follows 18 : 

[2] TE = Y j P j (J3 t +P ]+h ) + P r A ; where 

j = i 

n j 

P is the proportion of treatment group members in the j th stratum (i.e. — where N is the 

N ' 

total number of awardees in the sample and nj is the number of awardees in the j th stratum) , 
and 


P r is the proportion of treatment group members in the reference stratum. 

Overall covariate-adjusted mean for non-awardees 

_ b-l 

[3] 

^ Adj Non Awardees X P j (A + Pj + 1 ) + P r Po ' where 
j = 1 

n j 

P 

7 is the proportion of treatment group members in the j th stratum (i.e. N # where N is the 
total number of awardees in the sample and nj is the number of awardees in the j th stratum) , 
and 


is the proportion of treatment group members in the reference stratum. 


Overall Covariate-Adjusted Mean for Awardees 


[4] 


1 AdjAwardeesr 


- v +TE 

AdjNonAwardees 


Stratum-specific treatment effect estimates can be aggregated to yield an overall impact estimate in a 
number of ways. The method chosen here— weighing the estimate for each stratum by the proportion of 
treatment group members in that stratum— is widely used (Morgan and Harding, 2006; Caliendo and 
Kopeinig, 2007). 
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Estimated coefficients from the impact model and the overall impact estimate are presented as well 
as their corresponding standard errors and p-values. Hence, for dichotomous outcomes, impact 
estimates are presented in the form of percentage points. For continuous outcomes, overall impact 
estimates in "effect size" units (e.g., Hedges' g) are also presented. The effect size for an impact 
estimate was calculated as: 


[5] 


ES 


TE 

PooIedSD 


Where 


TE is calculated as shown in Equation 2, and 


PooIedSD 

[ 6 ] 


l (N r -1)S? +(N C -l)Sg 
\ (N,-\) + (N c -\) 


Where 


= sample size of treatment group, 

N = sample size of comparison group, 

S 2 = variance of the outcome for treatment group (unadjusted), and 
S 2 = variance of the outcome for comparison group (unadjusted). 
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Exhibit C.6: Pre-Award Data Used To Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact Models 

Covariate 

Reason for Inclusion in Impact Model 

Data Source(s) 

Type 

Definition 

Number of years 
between EAPSI 
application and Jan 1, 
2010 

Control for time elapsed since applying for EAPSI 

NSF Extant Data 

Continuous 

Variable 

Number of years between EAPSI 
Application and Jan 1, 2010 

Highest degree obtained 
as of Oct 1, 2010 

Control for highest degree differences 

EAPSI Survey Item D1 

Categorical 

3= PhD or equivalent 
2= M.A./M.S. or equivalent 
1= B.A./B.S. or equivalent 

Imputation flag for 
highest degree obtained 
as of Oct 1, 2010 

Flag to represent missing data pattern for 
Highest Degree variable 

Abt Created 

Dichotomous 

l=imputed 
0=not imputed 

Gender 

Control for gender differences. Also 
characteristic was not balanced in block 3. 

EAPSI Applicant 
Survey: G1 

Dichotomous 

l=Female 

0=Male 

Under-represented 
minority status (URM) 

Control for any differences due to URM status 

EAPSI Applicant 
Survey: G2, G3 

Dichotomous 

l=Other race(s)/ethnicity 
0=Asian Only or White Only 

Underrepresented 
minority status 
imputation flag 

Flag to represent missing data pattern for URM 
variable. 

Abt Created 

Dichotomous 

l=imputed 
0=not imputed 

Disability status 

Control for any differences due to disability 
status. Also characteristic was not balanced in 
block 2 

EAPSI Survey: Item G5, 
G5a 

Dichotomous 

l=Disabled 
0=Not Disabled 

Imputation flag for 
disability status 

Flag to represent missing data pattern for 
disability variable 

Abt Created 

Dichotomous 

l=imputed 
0=not imputed 
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Exhibit C.6: Pre-Award Data Used To Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact Models 


| Covariate 

Reason for Inclusion in Impact Model 

Data Source(s) 

Type 

Definition ij 

Field of study 

Control for applicants field of study. Also 
characteristic was not balanced in block 2. 

EAPSI Applicant 
Survey: A7 

Dichotomous 

l=Biological, agricultural, or 
environmental life sciences 
0=AII other disciplines 


1= Physical and related sciences 
(includes Chemistry except 
biochemistry; earth, 
atmospheric, ocean sciences; 
physics) 

0=AII other disciplines 
1= Computer and information 
sciences & Mathematics and 
statistics 

0=AII other disciplines 

1= Psychology & social sciences 
&related science 
0=AII other disciplines 


1= Engineering 
0=AII other disciplines 


1= Health (non-STEM) 
0=AII other disciplines 

Graduate degree 
program at time of 
application 

Control for any differences due to degree at time 
of application. Also characteristic was not 
balanced in block 1, 6. Only in models that used 
PSA stratification. 

EAPSI Applicant 
Survey: A5 

Categorical 

l=Master's 

2=Doctorate 

3=Other 

Prior visit to host 
location 

Characteristic was not balanced in block 1. Only 
in models that used PSA stratification. 

EAPSI Survey: Item C3 

Categorical 

0=No prior visit 

l=Prior visit to another site 

2= Prior visit to EAPSI application 

Site 
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Exhibit C.6: Pre-Award Data Used To Construct Comparable Groups of EAPSI Fellows and Unfunded Applicants for Impact Models 


Covariate 

Reason for Inclusion in Impact Model 

Data Source(s) 

Type 

Definition 

Prior relevant 
language(s) 

Characteristic was not balanced in block 5. Only 
in models that used PSA stratification. 

EAPSI Survey: Item C4 

Dichotomous 

l=prior language is language of 

EAPSI site 

0=else 

Total pre-award 
publications 

Control for prior number of publications used 
only in the model that examines post award 
publications 

EAPSI Applicant 
Survey: C8 

Continuous 

Total pre-award publication 

Percent publications with 
foreign collaborator (pre 
award) 

Characteristic was not balanced in block 3. Only 
in models that used PSA stratification. 

EAPSI Survey: Item C8 

Continuous 

Percent publications with 
foreign collaborator (pre award) 

Imputation flag for 
percent publications with 
foreign collaborator(pre 
award) 

Flag to represent missing data pattern for 
percent publications with a foreign collaborator 
variable. Only in models that used PSA 
stratification. 

Abt Created 

Dichotomous 

l=imputed 
0=not imputed 

Percent publications with 
foreign collaborator 
squared(pre award) 

Characteristic was not balanced in block 3. Only 
in models that used PSA stratification. 

Abt Created 

Continuous 

Percent publications with 
foreign collaborator squared(pre 
award) 
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Sensitivity Analysis 

A series of sensitivity analyses were conducted to ensure that the impact results reported are 
reliable and robust. In particular, the following sensitivity analyses were conducted: 

Using a different number of strata in the propensity score stratification 

As a sensitivity analysis, propensity blocks were combined and the impact estimate was recalculated 
using impact model [ 1] above. Blocks 1 and 2 were combined (blocks with a small number of 
treatment group members relative to the number of comparison group members) so we had a final 
total of five blocks instead of six. 

Check sensitivity to matching method used 

Stratification (also called interval matching) was the primary method for matching fellows and 
unfunded applicants. To check the sensitivity of the estimate to the matching method used, impacts 
were also estimated using the following propensity score matching methods: one-to-one (1:1), K-to- 
one (3:1), and radius (caliper) matching. 19 In its simplest form, 1:1 matching selects for each treated 
individual(s) the control individual with the closest propensity score (matching each fellow(s) with 
the most similar non-fellow). K: 1 and radius matching improves on 1:1 matching by restricting the 
distance of the matches. All of the matches are then pooled into matched treated and control 
groups and analyses were run using groups as a whole rather than as individual matched pairs. 

These methods discard treatment/comparison cases without matches which could potentially lead 
to a reduction in power 20 but could lead to higher precision. For the K:1 and radius matching weights 
were used in the analysis to represent the paring. The following model was then used to estimate 
the impact for this matching method: 


K is the outcome of interest for individual i, 

7] is the treatment indicator for individual i (l=awardee , O=non-awardee), 

X" is the it' 1 ' (n=l, 2,..., N) covariate for individual i (such as gender, age, etc.) that are grand- 
mean centered, and 

£ i is the usual error term for individual i. 

19 For the radius match a caliper of 0.005 was used. 

20 The reduction in power is minimal, for two reasons. First, in a two sample comparison of means, the 
precision is largely driven by the smaller group size (Cohen, 1988). So if the treatment groups stays the 
same size and only the control group decreases size, the overall power may not actually be reduced very 
much (Ho et al ., 2007). Second, the power increases when the groups are more similar because of the 
reduced extrapolation and higher precision that is obtained when comparing groups that are similar 
versus groups that are different (Snedecor, G. W. and Cochran, W. G. 1980. Statistical methods, 7th ed. 
Ames, IA: Iowa State University Press. MR0614143). 
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P 0 is the covariate-adjusted mean value of the outcome for the non-awardees, 

f5 j is the overall impact estimate (i.e. the difference between the mean value of the outcome of 
the awardees and non-awardees ), 

P 0 + (J 1 is the covariate-adjusted mean value of the outcome for the awardees, and 

Pn+(b-Y) (n=l,2,...,N) is the estimated overall relationship between the n covariate and the 
outcome controlling for other covariates. 

Sensitivity Checks Conclusion 

Sensitivity checks were conducted to determine how confident one could be with the results from 
the primary method of matching i.e. to determine how sensitive the findings are to the different 
matching methods used. 

Exhibit C.7 shows the overall standardized impact estimate and a 95 percent confidence interval for 
the estimate for each of the outcomes from the primary method of matching. It also shows the 
estimate and 95 percent confidence interval for the impact estimate for each of the outcomes for 
the other matching techniques. The impact estimates are very similar across the primary method, 
the 5 blocks, 3 to 1 and radius matching methods. The estimates and corresponding confidence 
interval from the 1 to 1 matching were sometimes different from the other methods but this was 
expected. In this method the "nearest comparison neighbor" can have a propensity score that is 
very different than the propensity score of its match. The differences in propensity scores of the 
resulting matched pairs in this method are as high as 0.880. Also, over 50 percent of the matches 
have differences over 0.328. 

The exhibit also shows that the estimates from the primary matching method are sometimes less 
precise (larger confidence intervals) then the estimates from other matching techniques. This is due 
to larger standard errors that occur because of the imbalance in the number of treatment and 
control members in some of the matched blocks in the primary method. Results from the 3:1 
method are reported since this increases precision while maintaining close propensity score 
matches and retains the most respondents after stratification. This method however could be 
criticized for not being externally valid since we don't know if the results could be generalized to the 
respondents that are excluded. However, the impact estimates from this method are similar to the 
impact estimates from the other matching methods. 
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Exhibit C.7: Overall Impact Estimates and Estimates with Each Sensitivity Analysis 


Had 2 or more Total Postdocs 


Recvd International Awards, etc. 


£ 

CO 



05 

CO 



Currently Empl as Res. Faculty 


Currently has Faculty Rank 


■o 

05 


05 

CO 



05 

CO 



Currently has tenure 


Total number of post-award publications 



Primary 5 block Itol 

3to1 

radius 

Primary 5 block Itol 

3to1 

radius 

Method 



Method 
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Total # of Inti. Postdocs Work with Individuals in Other Countries 




# of Pubs w/ Foreign Collab. 


Prop, of Pubs w/ Foreign Collab. 


0 

co 

E 


■Q 

0 

N 

'~o 

CO 

■a 

c 

CO 

CO 



£ 

co 

E 


■Q 

0 

N 

'~o 

CO 

■a 

c 

CO 

CO 



Emply. Outside the U.S. since EAPSI Mentored Others Traveling to Do Res. 



Primary 5 block Itol 

3to1 

radius 

Primary 5 block Itol 

3to1 

radius 

Method 



Method 
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Leadership in Fostering Inti Collab. Type of current work with foreign individual; 



Primary 5 block Itol 

3to1 

radius 

Method 





Primary 5 block Itol 

3to1 

radius 

Method 




EXHIBIT SHOWS: On the far left of each graph, the impact estimate and 95% confidence interval is shown for the primary 
matching method (stratification). Moving from left to right, the graphs show the impact estimate and 95% confidence 
interval for the different matching methods. 

NOTE: If the circle is above the dashed line representing 0.0 then the treatment effect is positive, if below 0 then the 
treatment effect is negative. If the confidence interval does not include 0 then the finding is significant. 
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Appendix D: Detailed Description of Benchmarking Analysis 


This appendix describes the methodology used to compare the EAPSI applicants and awardees and 
applicants to those of a nationally-representative sample. 

The 2006 and 2008 Survey of Doctoral Recipients (SDR) was utilized in this study to compare the 
EAPSI fellows and applicants, whose highest degree is a doctorate, to those of a nationally- 
representative sample of science, engineering and health (SEH) doctoral degree recipients on key 
employment, postdoctoral appointment, and international collaboration variables. The 2006 and 
2008 National Survey of Recent College Graduates (NSRCG) was utilized in this study to compare the 
EAPSI fellows and applicants, whose highest degree is masters, to a nationally-representative 
sample of SEH master's degree recipients on key employment, and international collaboration 
variables. 

For this study, the primary comparison group for EAPSI fellows is a propensity-score matched 
sample of unfunded EAPSI applicants. The SDR and NSRCG respondents were used as a secondary 
comparison group to assess how fellows' and all applicants' outcome indicators compare to national 
averages. SDR and NSRCG indicators most relevant to the study include employment sector and 
current position; the nature and extent of collaboration with foreign researchers. For example, 
comparisons included the percentage of respondents employed by academic institutions in the 
sample of EAPSI fellows and applicants versus the national samples (Question D2a in 2008 SDR). 

Exhibit D.l shows the applicant survey items that were designed to allow comparison of EAPSI 
awardees to NSRCG respondents. 

Exhibit D.2 shows the applicant survey items that were designed to allow comparison of EAPSI 
awardees to SDR respondents. 
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Exhibit D.l: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 National 
Survey of Recent College Graduates (NSRCG) 


EAPSI 

Applicant 

Survey 




Item 

NSRCG Item 

Item text 

Analysis variable 

D4 

B23 item 4, 

Work with individuals located in 

** EAPSI ** 


NSRCG 2006 

other countries? 

If D4=l then var=l; 


If D4=0 then var=0; 

**RCG** 

If B23 item 4=1 then var=l; 

If B23 item 4=0 then var=0; 

If A5=l & A6=l & A7=3 then var=. 


Control Variables: 

Field of study for the first 
Masters 

Years since degree 

- URM 
Gender 

D2 B12, NSRCG Was your primary employer 1 ** EAPSI ** 

2008 during the week of [reference If D2=l then var=l; 

week] 2 an educational institution? If D2=2 then var=0; 

**RCG** 

If B12=l then var=l; 

If B12=2 then var=0; 

If A10=l & All=l & A12=3 then var=. 

Control Variables: 

Field of study for the first 
Masters 

- URM 
Gender 

Employment Type: 

**EAPSI** 

Variables coded as: 
l=Yes, 0=No 

Report percent answering 'Yes' (1) to 
the following positions: 

Self-Employed or a Business 
Owner 

If D2 =0 and D3=l then var=l; 
0=else (note D2=l is included 
in this else) 

Private Sector 

If D2 =0 and D3=2 then var=l; 
0=else (note D2=l is included 
in this else) 

Local Government or State 
Government 


D3 Bll Which one of the following best 

NSRCG 2008 describes your primary employer 
during [reference week]? 
Self-Employed or a Business 
Owner 

Private Sector 

Local Government 

State Government 

U.S Federal Government 

U.S Military 

Other 
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Exhibit D.l: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 National 
Survey of Recent College Graduates (NSRCG) 


EAPSI 




Applicant 




Survey 




Item 

NSRCG Item 

Item text 

Analysis variable 


If D2 =0 and D3 in (3,4) then 
var=l; 0=else (note D2=l is 
included in this else) 


U.S Federal Government or 
U.S Military 

If D2 =0 and D3 in (5,6) then 
var=l; 0=else (note D2=l is 
included in this else) 

Other 

If D2 =0 and D3 in (7) then 
var=l; 0=else (note D2=l is 
included in this else) 

**NSRCG** 

Variables coded as: 
l=Yes, 0=No 

Report percent answering 'Yes' (1) to 
the following positions: 

Self-Employed or a Business 
Owner 

Bll in (1, 2) & B12=2 then 
var=l; 

Bll in (3, 4, 5, 6, 7, 8, 9) & 
B12=2 then var=0; 

If A10=l & All=l & A12=3 
then var=. 

Private Sector 
Bll in (3, 4) & A12=2 then 
var=l; Bll in (1, 2, 5, 6, 7, 8, 
9) & B12=2 then var=0; 

If A10=l & All=l & A12=3 
then var=. 

Local Government 

Bll in (5, 6) & B12=2 then 

var=l; 

Bll in (3, 4, 1, 2, 7, 8, 9) & 
B12=2 then var=0; 

If A10=l & All=l & A12=3 
then var=. 

U.S Federal Government or 
U.S Military 

Bll in (7, 8) & A12=2 then 
var=l; 

Bll in (3, 4, 1, 2, 5, 6, 9) & 
B12=2 then var=0; 

If A10=l & All=l & A12=3 
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Exhibit D.l: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 National 
Survey of Recent College Graduates (NSRCG) 


EAPSI 




Applicant 




Survey 




Item 

NSRCG Item 

Item text 

Analysis variable 


then var=. 


Other 

Bll in (9) & B12=2 then var=l; 
Bll in (3, 4, 1, 2, 1 , 8, 9, 5,6) 

& B12=2 then var=0; 

If A10=l & All=l & A12=3 
then var=. 


Control Variables: 

Field of study for the first 
Masters 

Years since degree 
- URM 
Gender 

NOTES: 

1 The EAPSI question asks about primary employer and NSRCG asks about principle employer. 

2 The reference week for the 2006 NSRCG was April 1, 2006; for the 2008 NSRCG was October 1, 2008; for the EAPSI 
applicant survey, October 1, 2010. 
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Exhibit D.2: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 Survey of 
Doctoral Recipients 


EAPSI 

Applicant 

Survey 




Item 

SDR Item 

Item text 

Analysis Variable 

Die 

A1 

SDR 2008 

During the week of [reference 
week] 1 were you working for pay 
or profit? (includes postdoctoral 
appointments) 

Working during reference week. 
**EAPSI** 

If Dl=3 & Die in (1,2) then var=l; 
If Dl=3 & Die in (3,4) then var=0; 


D2 A12 Was your primary employer 

SDR 2008 during the week of [reference 

week] an educational institution? 


**SDR** 

If Al=l then var=l; 

If Al=2 then var=0; 

Report percent answering 'Yes' (1). 

Control Variables: 

The number of years between 
receipt of first PhD degree and the 
reporting year of outcomes 
Field of study for the first PhD 

- URM 
Gender 

**EAPSI** 

If Dl=3 & D2=l then var=l; 

If Dl=3 & D2=2 then var=0; 


**SDR** 

If A12=lthen var=l; 

If A12=2then var=0; 

Report percent answering 'Yes' (1). 


Control Variables: 

The number of years between 
receipt of first PhD degree and the 
reporting year of outcomes 
Field of study for the first PhD 
URM 
Gender 


D3 

All 

Which one of the following best 

Employment Type: 


SDR 2008 

describes your primary employer 

**EAPSI** 



during [reference week]? 

Variables coded as: 



Self-Employed or a Business 

l=Yes, 0=No 



Owner 

Report percent answering 'Yes' (1) to 



Private Sector 

the following positions: 



Local Government 

Self-Employed or a Business 



State Government 

Owner 



U.S Federal Government 

If Dl=3 & D2 =0 and D3=l 



U.S Military 

then var=l; 0=else (note D2=l 



Other 

is included in this else) 




Private Sector 
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Exhibit D.2: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 Survey of 
Doctoral Recipients 


EAPSI 

Applicant 

Survey 

Item SDR Item Item text Analysis Variable 



If Dl=3 & D2 =0 and D3=2 
then var=l; 0=else (note D2=l 
is included in this else) 

Local Government or State 
Government 

If Dl=3 & D2 =0 and D3 in 
(3,4) then var=l; 0=else (note 
D2=l is included in this else) 
U.S Federal Government or 
U.S Military 

If Dl=3 & D2 =0 and D3 in 
(5,6) then var=l; 0=else (note 
D2=l is included in this else) 
Other 

If Dl=3 & D2 =0 and D3 in (7) 
then var=l; 0=else (note D2=l 
is included in this else) 

**SDR** 

Variables coded as: 
l=Yes, 0=No 

Report percent answering 'Yes' (1) to 
the following positions: 

Self-Employed or a Business 
Owner 

All in (1, 2) & A12=2 then 
var=l; 

All in (3, 4, 5, 6, 7, 8, 9) & 
A12=2 then var=0; 

Private Sector 

All in (3, 4) & A12=2 then 

var=l; All in (1, 2, 5, 6, 7, 8, 

9) & A12=2 then var=0; 

Local Government 

All in (5, 6) & A12=2 then 

var=l; 

All in (3, 4, 1, 2, 7, 8, 9) & 
A12=2 then var=0; 

U.S Federal Government or 
U.S Military 

All in (7, 8) & A12=2 then 
var=l; 

All in (3, 4, 1, 2, 5, 6, 9) & 
A12=2 then var=0; 

Other 
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Exhibit D.2: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 Survey of 
Doctoral Recipients 


EAPSI 

Applicant 

Survey 




Item 

SDR Item 

Item text 

Analysis Variable 




All in (9) & A12=2 then 
var=l; All in (3, 4, 1, 2, 7, 8, 
9, 5, 6) & A12=2 then var=0; 

D2a 3 

A14 3 

SDR 2008 

Currently employed as research 
faculty at Two-year college, 
community college or technical 
institute, 4-year 
college/university, medical 
school, or university-affiliated 

Currently employed at: 

**EAPSI** 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_3=l 
then var=l 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_3=0 
then var=0 


research 

**SDR** 

If A13 in (2, 3,4,5, 6) & A14_3=l then 
var=l 

If A13 in (2, 3,4,5, 6) & A14_3=0 then 
var=0 

Report percent answering 'Yes' (1). 

Control Variables: 

The number of years between 
receipt of first PhD degree and the 
reporting year of outcomes 
Field of study for the first PhD 
URM 
Gender 


D2b 4 

A14 4 

During the week of [reference 

Academic Position 


SDR 2008 

week] what type of academic 

**EAPSI** 



position did you hold at this 

Variables coded as: 



institution? 5 

l=Yes, 0=No 



President, Provost, or 

Report percent answering 'Yes' (1) to 



Chancellor (any level) 

the following positions:President, 



Dean (any level), department 

Provost, or Chancellor (any level) 



head or chair 

If Dl=3 & D2a in (2, 3,4, 5, 6) & D2b_l=l 



Research faculty, scientist, 

then var=l 



associate or fellow 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_l=0 



Teaching faculty 
Adjunct faculty 

then var=0 



Other (please specify): 

Dean (any level), department head or 
chair 




If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_2=l 
then var=l 

If Dl=3 & D2a in (2, 3,4, 5, 6) & D2b_2=0 
then var=0 




Research faculty, scientist, associate or 
fellow 
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Exhibit D.2: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 Survey of 
Doctoral Recipients 


EAPSI 

Applicant 

Survey 

Item SDR Item Item text Analysis Variable 



If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_3=l 
then var=l 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_3=0 
then var=0 

Teaching faculty 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_4=l 
then var=l 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_4=0 
then var=0 

Adjunct faculty 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_5=l 
then var=l 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_5=0 
then var=0 

Other (please specify) 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_6=l 
then var=l 

If Dl=3 & D2a in (2, 3, 4, 5, 6) & D2b_6=0 
then var=0 

**SDR** 

Variables coded as: 
l=Yes, 0=No 

Report percent answering 'Yes' (1) to 
the following positions: 

President, Provost, or Chancellor 
(any level) 

If A13 in (2, 3, 4, 5, 6) & A14_l=l then 
var=l 

If A13 in (2, 3,4,5, 6) & A14_l=0 then 
var=0 

Dean (any level), department head or 
chair 

If A13 in (2, 3,4,5, 6) & A14_2=l then 
var=l 

If A13 in (2, 3, 4, 5, 6) & A14_2=0 then 
var=0 

Research faculty, scientist, associate or 
fellow 

If A13 in (2, 3, 4, 5, 6) & A14_3=l then 
var=l 

If A13 in (2, 3, 4,5, 6) & A14_3=0 then 
var=0 
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Exhibit D.2: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 Survey of 
Doctoral Recipients 


EAPSI 




Applicant 




Survey 




Item 

SDR Item 

Item text 

Analysis Variable 


Teaching faculty 

If A13 in (2, 3, 4, 5, 6) & A14_4=l then 
var=l 

If A13 in (2, 3, 4, 5, 6) & A14_4=0 then 
var=0 

Adjunct faculty 

If A13 in (2, 3, 4, 5, 6) & A14_5=l then 
var=l 

If A13 in (2, 3,4,5, 6) & A14_5=0 then 
var=0 

Other (please specify) 

If A13 in (2, 3,4,5, 6) & A14_6=l then 
var=l 

If A13 in (2, 3,4, 5, 6) & A14_6=0 then 
var=0 

Control Variables: 

The number of years between 
receipt of first PhD degree and the 
reporting year of outcomes 
Field of study for the first PhD 
- URM 
Gender 
Faculty Rank: 

**EAPSI** 

If Dl=3 & D2c in (3,4,5) then var=l 
If Dl=3 & D2c in (6,7,8) then var=0 

**SDR** 

If A15 in (3,4,5) then var=l 
If A15 in (6,7,8) then var=0 

Report percent answering 'Yes' (1). 

Control Variables: 

The number of years between 
receipt of first PhD degree and the 
reporting year of outcomes 
Field of study for the first PhD 
URM 
Gender. 

Tenure Status: 

**EAPSI** 

If Dl=3 & D2d=3 then var=l 
If Dl=3 & D2d in (4,5) then var=0 


D2d 4 A16 4 What was your tenure status? 

SDR 2008 


D2c A15 Currently has faculty rank of 

SDR 2008 Assistant, Associate or Full 
Professor 
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Exhibit D.2: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 Survey of 
Doctoral Recipients 


EAPSI 




Applicant 




Survey 




Item 

SDR Item 

Item text 

Analysis Variable 


**SDR** 

If A16=3 then var=l 
If A16 in (4,5) then var=0 

Report percent answering 'Yes' (1). 

Control Variables: 

The number of years between 
receipt of first PhD degree and the 
reporting year of outcomes 
Field of study for the first PhD 
- URM 
Gender 
** EAPSI ** 

If D4=l then var=l; 

If D4=0 then var=0; 

**SDR** 

If A27 item 4=1 then var=l; 

If A27 item 4=0 then var=0; 

Report percent answering 'Yes' (1). 


D4 A27, Item 4 In performing the primary job you 

SDR 2006 held during the week of 

[reference week], did you work 
with individuals located in 
countries other than the US? 


Control Variables: 

The number of years between 
receipt of first PhD degree and the 
reporting year of outcomes 
Field of study for the first PhD 
URM 
Gender 


D6, item 1 

Cl, Item 2 
SDR 2008 

EAPSI: Peer-reviewed Journal 
articles 

SDR: Articles, (co)authored by 
you, have 

been accepted for publication in a 
refereed professional journal? 

Number of journal articles published 

**EAPSI** 

Dl=3 then 

Var=number of articles(0,l,2, ) 

**SDR** 

Var=number of articles(0,l,2, ) 




Control Variables: 

Field of study for the first PhD 
- URM 
Gender 

The number of years between 
application to EAPSI and survey 
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Exhibit D.2: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 Survey of 
Doctoral Recipients 


EAPSI 




Applicant 




Survey 




Item 

SDR Item 

Item text 

Analysis Variable 


reference date. Everyone in the 
SDR sample reported publications 
in the last 5 years. 


D6, item 2 


Cl, item 1 
SDR 2008 


EAPSI: Peer-reviewed conference 
publications (e.g. abstracts, 
conference papers, posters) 

SDR: Papers have you 
(co)authored for 

presentation at regional, national 
or 

international conferences? (Do 
not count presentations of the 
same work more than once.) 


Number of conference publications 

**EAPSI** 

Dl=3 then 

Var=number of conference 
publications^, 1,2, ) 

**SDR** 

Var=number of conference 
publications^, 1,2, ) 

Report percent answering 'Yes' (1). 


Control Variables: 

Field of study for the first PhD 

URM 

Gender 

EAPSI only: The number of years 
between application to EAPSI and 
survey reference date. Everyone in 
the SDR sample reported 
publication in the last 5 years. 


D6, item 3 C2, C3, item 1 

EAPSI: Patents, registered or 

Number of patents 

and 2 

pending 

**EAPSI** 

SDR 2008 


Dl=3 then 


SDR: Since October 2003, have 
you been named as an 

Var=number of patents(0,l,2, ) 


inventor on any application for a 

**SDR** 


U.S. patent? 

If C2=2 then var=0 


or 

How many applications for U.S. 

Else Var=sum(C3.1:C3.2) 


patents have named you as an 

Control Variables: 


inventor? 

Field of study for the first PhD 


or 

- URM 


Flow many U.S. patents have 

Gender 


been granted to you as an 

EAPSI only: The number of years 


inventor? 

between application to EAPSI and 
survey reference date. Everyone in 
the SDR sample reported 
publication in the last 5 years so 
this was not a control. 


D6, item 4 Cl, item 3 EAPSI: Book Chapter(s) Number of book chapter(s) 

SDR 2008 **EAPSI** 
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Exhibit D.2: EAPSI Applicant Survey Items Comparable to Items on the 2006/2008 Survey of 
Doctoral Recipients 


EAPSI 




Applicant 




Survey 




Item 

SDR Item 

Item text 

Analysis Variable 


SDR: Books or monographs, Dl=3 then 

(co)authored Var=number of chapters(0,l,2, ) 

by you, have been published or 

accepted for publication? **SDR** 

Var=number of chapters(0,l,2, ) 


Control Variables: 

Field of study for the first PhD 

URM 

Gender 

EAPSI only: The number of years 
between application to EAPSI and 
survey reference date. Everyone in 
the SDR sample reported 
publication in the last 5 years. 

NOTES: 

1 The reference week for the 2006 SDR was April 1, 2006; for the 2008 SDR was October 1, 2008; for the EAPSI applicant 
survey, October 1, 2010. 

2 The EAPSI question asks about primary employer and SDR asks about principle employer. 

3 For EAPSI, D2a is presented only if D2=Yes. For the 2006 SDR, Item A13 applies only if A12=Yes. 

4 For EAPSI, this item is only presented if D2=Yes and D2a not equal to preschool, elementary, middle, secondary school; 
for the 2006 SDR, this item applies only if A12=yes and A13 not equal to preschool, elementary, middle, secondary school 

5 Items have different response options. In EAPSI we collapsed Research Assistant, Teaching Assistant, and Postdoc into 
"other." Postdocs would skip out of D2b, so EAPSI did not have options for Research Assistant, or Teaching Assistant. 

For the SDR data we grouped Research Assistant, Teaching Assistant into the Other category. 


Limitations of These Comparisons 


There were four noteworthy limitations of these data. First, the SDR and NSRCG surveys were not 
designed to measure many of the outcomes that are pertinent to this study. Research productivity, 
for example, is a particularly notable omission. The use of the SDR and NSRCG as a national 
comparison is limited to the subset of items such as current employment, number of international 
collaborations, and a few others. 

The second limitation of the SDR and NSRCG data is the difference in timing, as this study collected 
data in 2011, whereas the SDR and NSRCG data come from surveys administered in 2006 and 2008. 
As a result, when the year of first doctoral or masters degree attainment was used as a variable on 
which to compare the study population to the SDR or NSRCG population, the groups are out of 
phase by two to five years, depending on the SDR or NSRCG cycle. For example, 2006 SDR 
respondents who earned their PhDs in 2000 would have had six years to achieve their outcomes by 
the time of the data collection. In contrast, respondents to the EAPSI survey who earned their PhD 
in the same year (2000) would have had 11 years to achieve outcomes, possibly biasing any 
comparison of the two groups. To mitigate this problem, comparisons controlled for the number of 
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years between the receipt of first PhD or first Master's degree and the reporting year of outcomes in 
the analyses. 

The third limitation is that the sampling frame for the SDR and NSRCG excludes individuals living 
outside of the United States during the survey reference period. This methodology creates a sample 
bias relevant to the EAPSI program. The sampling bias is compounded in that individuals excluded 
from the initial SDR/NSRCG would be excluded from any subsequent follow-up SDR/NSRCG surveys. 
As a result, the SDR/NSRCG sample might include fewer individuals likely to be engaged in 
international research collaboration. 

Finally, the SDR and NSRCG samples have the potential to include some EAPSI recipients. It was not 
possible to obtain personal identifying information for SDR or NSRCG respondents, so EAPSI 
participants could not be removed from the SDR or NSRCG sample. The 2006 SDR sample consisted 
of 30,817 individuals. The 2008 SDR sample consisted of 29,974 individuals. The 2006 NSRCG sample 
of master recipients consisted of 5,247 individuals. The 2008 NSRCG sample consisted of 5,442 
individuals. Since there are only 327 EAPSI awardees whose highest degree is a Master's and 501 
EAPSI awardees whose highest degree is a PhD in the sample, the potential overlap is very small 
(<1%) hence these were treated as independent samples. 


Performing the Calculations 


The goal of the benchmarking calculation was to perform a test of whether the difference between 
the two adjusted means from the two samples is equal to zero. Exhibit D.2 shows a specification of 
the hypothesis test and the method for calculating the p-value from the test is shown. Exhibit D.3 
shows a method for calculating the variance (and standard error) of the difference of the means and 
Exhibit D.4 shows a method for calculating the variance (and standard error) of the difference of the 
proportions (Survey Sampling by L Kish.). 

Exhibit D.2: Hypothesis Testing 

Let X t denote the estimated mean from the IFRP sample of size n x . 

X 2 denote the mean from the SDR sample of size fl 2 . 

SE(x 2 — JCj) denote the standard error of the difference between the two sample means. 

H a : x x - x 2 = 0 vs H a : x 1 - x 2 ^ 0 

Test Statistic is: 


(x l -x 2 )-0 
SE(x j - x 2 ) 

If the observed value of t as calculated above is greater than the critical value from the t-distribution with 
n -2 degrees of freedom and a=0.05, the null hypothesis will be rejected at the p<0.05 level. 

OR 


Abt Associates Inc. 


Appendix D D-13 




Evaluation of the East Asia and Pacific Summer Institutes Program: Final Report 


V Yl 
Let 1 1 denote the estimated proportion from the IFRP sample of size 

V Yl 
denote the estimated proportion from the SDR sample of size 2 . 

SE(p { — p 2 ) denote the standard error of the difference between the two sample proportion. 

H 0 ■ Pi - p 2 = 0 vs H a : Pl -p 2 * 0 

Let the z be the standardized deviate, calculated as: 

_ Oi -p 2 )-° 

SE{p { -p 2 ) 

Compare z to the quantiles of a standard normal distribution, N(0,1), to find the two-sided probability of 
obtaining a deviate with absolute value that is as large or larger than z. If the absolute value of z is greater 
than 1.96, the null hypothesis will be rejected at the p<0.05 level. 
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Exhibit D.3. Estimating the Variance of the Difference in Two Sample Means 


Let Xj denote the estimated mean from the IFRP sample of size . Let X 2 denote the mean from the SDR 

sample of size Tl 0 . The difference between the two sample means was tested. The estimated variance of 
the difference between the two sample mean is written as 


v(Xj - x 0 ) = v(Xj ) + v(x 9 ) - 2cov(x 1 , X 2 ) 

Linder simple random sampling, the variance of the difference can be written as 

2 P. V2 m ^ v (x iMx 2 ) 


v(Xj - x 2 ) = v(Xj ) + v(x 2 ) - ■ 


V'V 


1 ^ is the estimated variance of the mean based on the EAPSI sample of units, 1 | s the estimated 

Tl 

variance of the SDR mean based on 2 units and m is the amount of overlap between the two samples. 

The correlation ( ^ x ' Xl ) is estimated based on the overlap. Since m is almost 0 the variance can be 
estimated as: 


v(Xj - x 2 ) = v(Xj ) + v(x 2 ) 

The variance under the sample 21 design is obtained from proc survey reg for the first mean and the second 
mean. The square root of the variance gives the standard error of the difference in the two means, which 
can be used in a statistical test. 


The EAPSI sample was treated as a simple random sample or "pseudo-sample” since awardees from a 
different grant period might result in a different estimate. 
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Exhibit D.4: Estimating the Variance of the Difference in Two Sample Proportions Based on 
Independent 22 Samples. 


Let Pi denote the estimated proportion from the IFRP sample of size P . Let P 2 denote the 

fl 

proportion from the SDR sample of size 2 . The difference between the two sample means 
was tested. The estimated variance of the difference between the two sample mean is written 
as 

v(P\ ~ P 2 ) = v (Pi ) + v(p 2 )~ 2mco\(p l ,p 2 ), 

Since m is almost 0 one can write the estimated variance of the difference between the two 
proportions as: 

v(p 1 ~p 2 ) = v(p l ) + v(p 2 ) 

One can get the variance under the sample 23 design from proc survey reg for the first 
proportion and the second proportion. The square root of the variance gives the standard error 
of the difference in the two means, which can be used in a statistical test. 


Estimation of Mean and Variance 


As mentioned above SAS proc survey reg was used to obtain the adjusted mean and standard error. 
The following regression model was used to estimate the adjusted mean and standard error for 
each sample: 

5’,=a+Za^,"+e, 

n = 1 


Where: 


Y ] is the outcome of interest for individual i, 

X" is the n' h (n=l, 2,..., N) covariate for individual i (such as gender, URM, number of years 
between PHD and survey date, etc.) that are grand-mean centered across the samples, and 

£, is the usual error term for individual i. 


Interpretation of Parameters 


Interpretation of the coefficients in the model is as follows: 


The EAPSI sample is a potentially a VERY small proportion (close to 0) of the SDR sample so these were 
treated as independent samples. 

The EAPSI sample is treated as a simple random sample or "pseudo-sample" since awardees from a 
different grant period might produce a different estimate. 
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[3 0 is the covariate-adjusted mean value of the outcome for the sample, 

1 3 n (n=l,2,...,N) is the estimated overall relationship between the n covariate and the outcome 
controlling for other covariates. 
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Appendix E: Surveys 
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East Asia and Pacific Summer Institutes 
(EAPSI) Program 

Applicant Survey Items 




Programming notes appear in red or brown text throughout. 

Hyperlinked text will appear tfflbosstdl iwd wdtertii^tdl in this document but should not be 
formatted this way onscreen unless noted (underlining is fine except for email addresses, but 
embossing is not). 

General guidelines: 

Each screen displayed one question at a time unless otherwise indicated. 

Each screen displayed a standard set of Navigaton buttons shown below. 


r 


■\ 


MODULE A: VERIFYING INFORMATION ABOUT YOU 

Based on the information contained in the NSF EAPSI database, you participated as a summer research fellow in the National 
Science Foundation's East Asia and Pacific Summer Institutes program (EAPSI) in the summer of 2006. Is this correct? If you 
participated in EAPSI more than once, we refer here, and throughout the rest of the survey, to vour most recent EAPSI experience 
unless specifically stated otherwise. 


© Yes, this is correct. 

© No, I never participated in EAPSI. 

© No, I appted for the EAPSI felovvship, but withdrew my application before the award decision had been made. 
© No, I was awarded the EAPSI fellowship, but declined it. 

No, I participated in EAPSI, but the most recent year I participated was (Enter year as yyyy): 



Some items contained placeholders for fields that were programmed to be auto-filled from the 
sample file for each respondent. These fields are enclosed in [brackets] and printed in brown. 

Textboxes for open-text responses are indicated with [textbox, 150] where the number indicates the length of 
the field (how many characters the respondent may type) 

REMINDERS for BLANK RESPONSES and DEFAULTS for SKIP PATTERNS if RESPONSE was 
MISSING 

• No items required a response. 

• No items required a reminder to enter an answer/select a response unless indicated 
below. If a reminder was needed for the first time user leaves an item blank, the item 
was re-displayed adding the text below shown on-screen in bold, red text. 


These reminders and defaults are given at the end of this document. 


Sample variables. 


AwdStatus: Fellow, Unfunded: Two groups of respondents will be tracked using the sample file: 
Fellows, and Unfunded Applicants. In the sample file, Awardee = FELLOW and Declinee = 
UNFUNDED APPLICANT. 

PI_Name will contain First Name, Middle Name, Last Name. Note that Middle Name may be an 
initial or may be blank. 

EAPSI_year: The year for which the applicant applied to participate in EAPSI 

EAPSI_app_site: The name of the site to which applicant applied (as first choice) 

EAPSI_actual_site: The site which a Fellow actually visited for the EAPSI fellowship 

Dropdown EAPSI site menu: When clicked and held this dropdown menu allows R to choose 
one of the following: 

[blank] 

Australia 

China 

Japan 

Korea 

New Zealand 

Singapore 

Taiwan 

Once a selection is made, the variable EAPSI_app_site or EAPSI_actual_site should be set to this 
value. 

University: The US graduate institution where an EAPSI applicant was enrolled 

CurrGrad = 1 if Item Flc = 3 after Respondent completes item (see below); CurrGrad = 0 
otherwise. 



Introductory screen: 


Survey of former applicants to the 
National Science Foundation’s 
East Asia and Pacific Summer Institutes (EAPSI) program 


Welcome and thank for your interest in this study. This survey is being conducted by Abt 
Associates Inc. and our subsidiary, AbtSRBI, for the National Science Foundation (NSF), to learn 
about former graduate students who applied to NSF's East Asia and Pacific Summer Institutes 
(EAPSI) program. This survey will give NSF information about the professional characteristics 
and international collaborations of U.S. scientists and engineers and help NSF improve programs 
intended to foster a globally engaged scientific and engineering workforce. You are receiving 
this survey because you are listed in the NSF database as a former applicant to EAPSI. We 
estimate that it will take approximately 30 minutes to complete the survey. 

NSF's records indicate that you applied for a summer graduate research fellowship in one of 
the following sites between 1999 and 2010: 


These graduate summer fellowships are part of NSF's East Asia and Pacific Summer Institutes 
program (EAPSI). If you do not recall applying to this program, please click here: 


Otherwise, please continue. The next page has important information about this study. 


Japan 

Korea 

Taiwan 

China 


Australia 
New Zealand 
Singapore 


I don*t racaM applying 


Continue 


Click on "Continue" directs R to Screen 2 
Click on "I don't recall" directs R to Confirm Fellowship screen 




Screen 2 


EAPSI Applicant Survey 


Confidentiality and Participation 

Participation in the survey is voluntary and nonparticipation will have no impact on you or your 
institution. You may skip questions on the survey or discontinue participation at any time. There 
are minimal risks associated with your participation. We take your privacy very seriously. Your 
responses to this survey will be protected under the Privacy Act. There is minimal risk of breach 
of confidentiality, and we have put in place procedures to minimize this risk. Reports will never 
identify you by name, and information from the study will only be reported in the aggregate at 
the program level, combined with about 500 other responses. When we receive your survey we 
will detach and store separately your name and other identifying information that could be used 
to link you to your survey responses. Survey responses will be stored on a secure drive that is 
only accessible to members on the study team. Only study team web technicians and data 
analysts from Abt Associates and AbtSRBI will see individual responses that can be linked to you. 
Survey data files will be shared with NSF at the end of the study, only after study team members 
have examined the data to be free of any information that could help identify you; this cleaning 
includes procedures to limit someone from inferring your identity by analyzing non-identifying 
data. Hence, we encourage you to respond candidly about your experiences. Separate from 
your individual responses to the survey we will provide NSF any updated contact information we 
have found or requested from you. None of this contact information will be linked in any way to 
your survey responses. At the conclusion of the study, Abt Associates and AbtSRBI will destroy 
all records, electronic or otherwise, that link you to your survey responses. 


Questions 

If you have questions about the study, please contact the study director, Alina Martinez of 
Abt Associates Inc. at (866) 421-6223 (toll free within the U.S.) or email her at 
EAPSI_survey@abtassoc.com. You may also contact the evaluation's program officer at NSF, 

John Tsapogas (tsapoga@nsf.gov). If you have questions about your rights as a research 
participant, you may contact Teresa Doksum, the Abt Institutional Review Board Administrator 
at (877) 520-6835 (toll free within the U.S.) or by email: irb@abtassoc.com. To learn more 
about this study, please refer to the FlWIUWtlif Aste^l page. 

Consent 

Please click on "Begin" if you agree to participate in this study. pGIN] 

Click on BEGIN takes R to Navigation Screen 


This study's IRB approval number is #0494, valid from 8/6/2010 to 8/5/2011. For questions, please 
contact Teresa Doksum, IRB Administrator, Abt Associates, at IRB@abtassoc.com. 

The valid 0MB control no. for this information collection is 3145-0214. (Expires on 12/31/13) 




Navigation screen: 


EAPSI Applicant Survey 


Navigating through the survey: 

As you work through the survey, your responses are automatically saved. You may change a 
response by clicking on the PREVIOUS ITEM button. Use theNEXTITEM button to advance to 
the next question. At any time, you may close your browser if you wish to return and finish at 
a later time. When you log back in, the survey will take you to where you left off. On each 
page of the survey, afAQ button is provided if you have a question during the survey or need 
information about how to contact the survey administrator. 

When you have completed the survey, please click on theSUBMIT button at the end of the 
survey. You may submit the survey even if there are some questions that you choose not to 
answer. 

NEXT 




FAQs are Optional screens, displayed only if R clicks on FreWfefttly Asted QUSlWm 


FREQUENTLY ASKED QUESTIONS 

% What is the East Asia and Pacific Smmrcvsr Institutes (EAPSI) Program? 

• Why are vov doing this study? 

% Why have I hew select to rerttetoate fa this swvfey? 

% Howclhlyouiatmyewtactfaloymatfaft? 

* HowteftewiMthissmiveytatetotomBtete? 

« HowwWyottvsemytommeiftts? 

% Does this study have human sohtetts review tharawt? 

% Who is ftuftdiire the study? 

% Who are Abt Associates Ire and AbtSRBI? 

What is the East Asia and Pacific Summer Institute (EAPSI) Program? 

The East Asia and Pacific Summer Institute (EAPSI) Program is a National Science Foundation 
(NSF) program that provides funding for U.S. graduate students to spend eight to ten weeks 
collaborating with a host researcher in one of seven sites in the East Asia and Pacific region: 
Australia, China, Japan, Korea, New Zealand, Singapore, and Taiwan. 

Why are you doing this study? 

NSF is interested in learning about the characteristics of current and former graduate students 
who applied to participate in the EAPSI program sometime between 1999 to 2009. 

Information about the domestic and international experiences of EAPSI participants and non- 
participants will help NSF understand the usefulness and relevance of international research 
training for scientists and engineers. Information from this study will be used to describe the 
characteristics and subsequent career paths of graduate students who expressed interest in 
pursuing an international collaboration. 

Why have I been selected to participate in this survey? 

You have been selected to participate because we have identified you as having applied to the 
EAPSI program while you were a graduate student. 

How did you get my contact information? 

We identified you from records maintained by the National Science Foundation on prior EAPSI 
applicants. We then obtained your contact information through an internet search. 

How long will this survey take to complete? 

We estimate that the survey will take about 30 minutes. 

How will you use my comments? 

Responses from all survey respondents will be used to answer questions about the graduate 
training, international experience, and career paths of applicants to the EAPSI program between 
1999 and 2009. 


Has this study been reviewed and approved by an institutional review board? 

Yes, the study was approved by Abt Associates' Institutional Review Board. If you have any 
concerns about your participation in this survey, please contact Teresa Doksum, Institutional 
Review Board Administrator at Abt Associates, at (877) 520-6835 or via email at 
irb@abtassoc.com. 

Who is funding the study? 

The study has been funded by the National Science Foundation under contract 
GS-10F-0086K. Abt Associates and AbtSRBI will complete the study. 

Who are Abt Associates Inc and AbtSRBI? 

Abt Associates is an independent research firm headquartered in Cambridge, Massachusetts. 
AbtSRBI is a wholly-owned subsidiary of Abt Associates specializing in large-scale data collection 
and public opinion research. NSF has contracted with Abt Associates and AbtSRBI to design and 
implement a survey of graduate students who applied to NSF's EAPSI program. 


EAPSI Applicant Survey 


Please note: In this survey, we will refer to East Asia and Pacific Summer Institute (EAPSI) 
"sites," or "locations" rather than EAPSI "countries." For example, Australia, China, Korea, 
Japan, New Zealand, Singapore, and Taiwan were EAPSI sites in 2010. 

MODULE A: VERIFYING INFORMATION ABOUT YOU 

To begin, we'd like to confirm that we've reached the appropriate person. 

Al. Based on the information that you provided to NSF in your application, your name is [First 
Name, Middle Name/initial, Last Name], Is this correct? 

□ No, my name has changed or my name is misspelled above. My name is: [textbox, 75] 

FELLOWS GOTO A2.1 

UNFUNDED APPLICANTS GOTO A2.2 

□ No, I'm not the person named above. -> EXIT SCREEN 

□ Yes, this is correct. 

FELLOWS GOTO A2.1 

UNFUNDED APPLICANTS GOTO A2.2 

Programmer note: text highlighted in yellow is new/changed 
[Former Fellows only]: 

A2.1 Based on the information contained in the NSF EAPSI database, you participated as a 

summer research fellow in the National Science Foundation's East Asia and Pacific Summer 
Institutes program (EAPSI) in the summer of [EAPSI_year], Is this correct? 

If you participated in EAPSI more than once, we refer here, and throughout the rest of the 
survey, to your most recent EAPSI experience unless specifically stated otherwise. 

□ Yes, this is correct. -> GOTO A3 

□ No, I never participated in EAPSI. CONFIRM FELLOWSHIP SCREEN Section R 

□ No, I applied for the EAPSI fellowship, but withdrew my application before the award 
decision had been made. EXIT SCREEN 

□ No, I was awarded the EAPSI fellowship, but declined it. ->EXIT SCREEN 

□ No, I participated in EAPSI, but the most recent year I participated was (Enter year as 
yyyy): [textbox, 4], GOTO A3 Set EAPSI_year = entered year. Prompt R to correct 
entry: "Please type a four-digit year between 1998 and 2010" 


[Unfunded applicants only] 

A2.2. Based on the information contained in the NSF EAPSI database, you most recently applied 
for a [EAPSI_YEAR] EAPSI summer research fellowship, but did not participate in this 
fellowship. Is this correct? 

□ Yes, this is correct. GOTO A3 

□ No, I did participate in an EAPSI fellowship: my fellowship took place in the summer of 

(Enter year as yyyy): [textbox, 4] [Prompt R to correct entry: "Please type a four-digit 
year between 1990 and 2010"]] in [Select locationsite: [dropdown EAPSI site menu], 
GOTO CONFIRM FELLOWSHIP SCREEN Section S Set EAPSI_year = entered year; Set 
EAPSI_actual_site = Selection 

□ No, I never applied for this program. CONFIRM FELLOWSHIP SCREEN Sections 




A3. Based on the information contained in the NSF database, when you most recently applied 
for an EAPSI fellowship, the first choice of host site you listed was [EAPSI_app_site]. (If you 
received an EAPSI fellowship, you may have visited a different site, but here we want to 
know what site you listed in your application as your first choice.) 

Is this correct? 

□ Yes, [EAPSI_app_site] was my first choice 

□ No, the first choice I listed on my most recent application for an EAPSI fellowship was: 
(Select site): [dropdown EAPSI site menu]. Set EAPSI_app_site to selection. 

□ I do not recall 

Fellows -> Goto A3a. 

Unfunded applicants -> Goto A4. 

A3a. [Fellows only]: Some EAPSI fellows did not receive their first choice of host site. In 
which site was your EAPSI fellowship? Select site: [dropdown EAPSI site menu] If you 
have held more than one EAPSI fellowship, please respond based on your most recent 
fellowship. Set EAPSI_actual_site = selection 

A4. Based on the information that you provided to NSF in your application, you were enrolled at 
[University] when you most recently applied for an EAPSI fellowship. Is this correct? 

□ Yes 

□ No, I was enrolled as a graduate student at (Enter full name of graduate instituition 
without abbreviations): [textbox, 100], Set UNIVERSITY = text entered 

A5. What graduate degree were you pursuing when you most recently applied for the summer 
EAPSI fellowship? 

□ Master's degree (MA, MS) 

□ Doctoral degree (PhD, EdD, MD, joint MD/PhD, PsyD, ScD) 

□ Other -specify: [textbox, 50] 

A5a. As of [October 1 st , 2010], had you earned this degree? 

□ Yes Go to A5b 

□ No -» Goto A6 

A5b. What month and year did you receive this degree? 

| (mm|yyyy) Prompt R to correct entry: "Please enter a two-digit 

month (or leave it blank) and a four-digit year (or leave it blank)." 


A6. In what month and year did you first enroll as a graduate student at [University]? Enter 

two-digit month and four-digit year: [ | (mm |yyyy)] Prompt R to correct entry: 

"Please enter a two-digit month (or leave it blank) and a four-digit year (or leave it 
blank)." 


A7. What discipline were you studying as a graduate student when you applied for an EAPSI 
fellowship? Please select the field that best matches your discipline. First, indicate which 
one of four broader areas in which your discipline falls: Check one only: 

□ i Sciences [no hyperlink] 

(e.g., Biological/Life, Chemical, Computer/Information, Environmental, 
Earth/Atmospheric/Oceanographic, Geosciences, Mathematical/Statistical, 
Physics, Psychology) 

□ 2 Social Sciences [no hyperlink] 

(e.g., Economics, Sociology, Anthropology/Archaeology, Political Science, 
Geography, Linguistics, other) 

□ 3 Health/Medical fields[no hyperlink] 

□4 Engineering [no hyperlink] 

display one of three screens based on choice above: 

1 Science screen 

2 Social Science, Health screen 

3 Social Science, Health screen 

4 Engineering screen 

The next three pages should each be a separate screen. After R hits NEXT, go to Item B1 


Science Fields 


For Science/Math Education select the primary scientific or mathematical field, or select 
SocW SctafWBS or and see "OTHER Social Sciences: 621: Education, 

General" 

To view other fields (Social Science, Health/Medical, Engineering), please select©© 

bttfc t© th© 4 bHHri categories to change your selection. 


Agricultural/food sciences 

O 

Animal sciences 

O 

Plant sciences 


O 

Food sciences/technology 

O 

OTHER agricultural sciences 

Biological sciences 

O 

Biochemistry/biophysics 

o 

Microbiological sciences/immunology 

Bioengineering, see 

O 

Biology, general 

o 

Nutritional sciences 

ENGINEERING, next page 

o 

Botany 

o 

Pharmacology, human/animal 


o 

Cell/molecular biology 

o 

Physiology and pathology, 


o 

Ecology 


human/animal 


o 

Genetics, animal/plant 

o 

Zoology, general 




o 

OTHER Biological sciences 

Environmental life sciences 

o 

Environmental science/studies 

o 

Forestry sciences 

Computer and information 

o 

Computer/information sciences, 

o 

Computer systems analysis 

sciences For Computer 


general 

o 

Information services/systems 

Engineering, see 

o 

Computer programming 

o 

OTHER computer/information sciences 

ENGINEERING, next page 

o 

Computer science 



Mathematics and statistics 

o 

Applied mathematics 

o 

Statistics 


o 

Mathematics, general 

o 

OTHER mathematics 


o 

Operations research 



Chemistry, except biochem 

o 

Chemistry except biochemistry (biochemistry, see Biological sciences) 

Earth, atmospheric, and 

o 

Atmospheric 

o 

Geological sciences, other 

ocean sciences 


sciences/meteorology 

o 

Oceanography 


o 

Earth sciences 

o 

OTHER physical sciences 


o 

Geology 



Physics Biophysics, see 

O Astronomy/astrophysics 

o 

Physics 

Biological Sciences 





Psychology 

O 

Clinical psychology 

o 

General psychology 


O 

Counseling 

o 

Industrial/organizational psychology 


O 

Educational psychology 

o 

Social psychology 


o 

Experimental psychology 

o 

OTHER psychology 


GO BACK TO 4 BROAD CATEGORIES 


NEXT ITEM 


Social Sciences and Health/Medical fields 


For Science/Math Education select the primary scientific or mathematical field (click on 
SdieifUC^i . or see below "OTHER Social Sciences: 621: Education, General" 

To view other fields (Science, Engineering), please selrr.t Go th»4 broad categories to 

change your selection. 


Economics 

O Agricultural economics O Economics 

Political and related 
sciences 

O International relations O Public policy studies 

O Political science/government 

Sociology/Anthropology 

O Anthropology/archaeology O Sociology 

O Criminology 

OTHER social sciences 

O Area/ethnic studies O Linguistics 

O Education, general O Philosophy of science 

O Geography O OTHER social sciences 

O History of science 

Health 

O Audiology/speech pathology O Nursing (4 years or longer program) 

O Health services administration O Pharmacy 

O Health/medical assistants O Physical therapy and other 

O Health/medical technologies rehabilitation/therapeutic services 

O Medical preparatory programs (e.g., O Public health (Including environmental 

pre-dentistry, pre-medical, health/epidemiology) 

pre- veterinary) O OTHER health/medical sciences 

O Medicine (e.g., general, internal, orthopedic, 

surgical, dentistry, optometry, osteopathic, 
podiatry, veterinary) 


GO BACK TO 4 BROAD CATEGORIES 


NEXT ITEM 


Engineering fields 

For Science, Math or Engineering Education select the primary scientific or 
mathematical field (click on Seteft^T or select Somali Sditflees Of Htalfth/MtdteiH and see 
"OTHER Social Sciences: 621: Education, General" 


To view other fields (Science, Social Science, Health/Medical), please select Gobadtto<hfe4 
htwritategwtes to change your selection. 


Biochemical engineering, see Bioengineering/biomedical under OTHER Engineering 

Chemical engineering 

O 

Chemical engineering 



Civil/architectural eng. 

O 

Architectural engineering 

O 

Civil engineering 

Electrical/computer 

o 

Computer/systems engineering 

O 

Electrical/electronics/communications 

engineering 




engineering 

Industrial engineering 

o 

Industrial/manufacturing engineering 



Mechanical engineering 

o 

Mechanical engineering 



OTHER engineering 

o 

Aerospace/aeronautical/ 

o 

Geophysical/geological engineering 



astronautical engineering 

o 

Materials engineering, including 


o 

Agricultural engineering 


ceramics/textiles 


o 

Bioengineering/biomedical 

o 

Metallurgical engineering 



engineering 

o 

Mining/minerals engineering 


o 

Engineering, general 

o 

Naval architecture/marine engineering 


o 

Engineering sciences/ 

o 

Nuclear engineering 



mechanics/physics 

o 

Petroleum engineering 


o 

Environmental engineering 

o 

OTHER engineering 


GO BACK TO 4 BROAD CATEGORIES 


NEXT ITEM 


MODULE B: THE EAPSI APPLICATION PROCESS 


Bl. Why did you apply for an EAPSI fellowship? Check all that apply. 

□ To conduct research with a specific person or at a specific institution 

□ To collaborate with a foreign scientist 

□ To understand what research in my field was like outside the US 

□ To access resources (e.g. samples, equipment) for research that I could not find in the 
U.S. 

□ To enhance my skills or knowledge as a researcher 

□ To make progress towards earning my graduate degree 

□ To enhance my resume as a future job candidate 

□ To learn about the culture, history, and/or geography of another place 

□ To learn another language 

□ To travel outside of the United States 

□ For family reasons (e.g., to accompany a spouse traveling abroad) 

□ Other (please specify): [textbox, 300] 

B2. For your application to the EAPSI program, why did you select the site you indicated was 
your first choice for host site? Check all that apply. 

□ The host researcher there was conducting research relevant to my own interests 

□ The host institution had equipment or resources helpful for my proposed research 

□ A faculty advisor or mentor recommended this particular site, the particular host 

institution or host researcher in this site 

□ I have professional ties with someone who is from this site 

□ The host researcher had visited my department or university 

□ There were particular places in this site (aside from the host institution) that I wanted to 

visit or see 

□ I had studied its culture, history, politics, geography, etc. before applying 

□ I was familiar with the primary language(s) spoken there 

□ Other reason (please specify): [textbox, 300] 


B3. Did the notification of your application status allow you sufficient time to make the 

necessary arrangements for your visit or to make alternative plans if you did not receive the 
award? Check one only. 

□ Yes 

□ No 

□ I don't recall 

B4. How did your primary graduate advisor view your decision to apply for an EAPSI? check 
one only. 

□ I do not know how my advisor viewed my decision to apply 

□ I did not have an advisor when I most recently applied 

□ My advisor opposed my decision to apply 

□ My advisor was indifferent to my decision to apply 

□ My advisor supported my decision to apply 

□ My advisor encourage me to apply to the EAPSI program 

B5. What types of mentoring or guidance did you receive from your graduate advisor or other 
mentors during the preparation of the EAPSI application? Check all that apply. 

□ Provided a letter of recommendation to NSF 

□ Suggested a host institution 

□ Recommended me to a colleague at the host institution 

□ Provided feedback on my project proposal 

□ Discussed cultural and language aspects of the host site with me 

□ Other -specify: [textbox, 300] 

□ None 

Fellows GO TO Item B6 
Unfunded Applicants SKIP to Item Cl 


B6. Did you participate in a pre-departure orientation? Check one only. 

□ Yes 

□ No-^Go To Cl 

□ I do not recall ->Go To Cl 


B6a. Did the pre-departure orientation provide adequate information on the following 
topics? Check all that apply. 

□ Living expenses 

□ Healthcare access 

□ Lodging 

□ Culture and rules of the host site 

□ Political situation in the host site 

□ What not to do in the host site 

□ What to do in case of emergency 

□ I do not recall->Go To Cl 

B6b. What additional topics do you think should have been covered during the orientation? 
[textbox, 500], 


MODULE C: ABOUT YOU AT THE TIME OF YOUR APPLICATION 

Items in this section ask for information about the year in which you applied for an EAPSI 
fellowship. 

Cl. Using a 4-point scale, what was your overall undergraduate grade point average (GPA)? 

If you have more than one bachelor's degree, give your overall grade point average for your 
first bachelor's degree. Check one only. 

□ 3.75 -4.00 GPA (Mostly A's) 

□ 3.25 - 3.74 GPA (About half A' s/half B's) 

□ 2.75 -3.24 GPA (Mostly B's) 

□ 2.25 - 2.74 GPA (About half B' s/half C's) 

□ 1.75 -2.24 GPA (Mostly C's) 

□ 1.25 - 1.74 GPA (About half C's/half D's) 

□ Less than 1.25 (Mostly D's or below) 

□ Did not take courses where grades were given 

C2a. While you were an undergraduate, did you participate in a study abroad program or did 
you spend a semester (or more) pursuing your education outside the United States? Check 
one only. 

□ Yes 

□ No 

C2b. Before applying to EAPSI, had you participated in a study abroad program as a graduate 
student or spend a semester (or more) pursuing your research or graduate education 
outside the United States? Check one only. 

□ Yes 

□ No 

C3. Prior to applying for an EAPSI fellowship, had you ever visited or lived in any of the following 
places for a period of 30 days or longer? Check all that apply. 

□ Australia 

□ New Zealand 

□ China 

□ Japan 

□ Korea 

□ Taiwan 

□ Singapore 

□ None of the above places 



C4. At the time you applied for an EAPSI fellowship, had you studied, or were you conversant in, 
any of the following languages? Check all that apply. 

□ Japanese 

□ Korean 

□ Chinese (Mandarin or Cantonese) 

□ Malay 

□ Tamil 

□ Maori 

□ Hindi or Urdu 

□ Russian 

□ None of the above 


C5. At the time you applied for an EAPSI fellowship, had you participated in any international or 
intercultural clubs, organizations, or professional associations? Check one only. 

□ Yes 

□ No 


C6. 


At the time you applied for EAPSI, had you done any of the following? Check all that apply 

□ Attended elementary or secondary school in another country (outside the U.S.) 

□ Lived outside the U.S. for six months or longer 

□ Collaborated on research with someone based in another country 

□ Published research with someone based in another country 

□ Attended or presented scholarly work at a research conference in another country 

□ Participated on a research team with a scientist who was visiting my graduate 
institution from a foreign institution 

□ Worked with a colleague who had completed an EAPSI Fellowship 

□ None of the above 


C7a. In your application for EAPSI, did you include a letter of support (or other written 
communication, such as an email message) from the host scientist with whom you 
proposed to work? Check one only. 

□ Yes 

□ No 

□ I do not recall 

C7b. At the time you applied for EAPSI, were you already collaborating with the host scientist 
or someone else at the host institution to which you applied? Check one only. 

□ Yes, with my proposed host scientist 

□ Yes, with someone else at the host scientist's institution 

□ No 



C7c. Which of the following were true at the time of application? Check all that apply. 

□ My graduate advisor or another faculty member in my department was 
collaborating or had collaborated with a researcher at the host institution 

□ My graduate department or university had an existing collaboration or graduate 
student exchange program with the host institution 

□ Other graduate students from my university or department had been to the host 
institution 

□ There was another type of relationship between my graduate institution and the 
host institution. Please specify: [textbox, 300] reminder for 1 st no entry in textbox 
entry in textbox is not required 

□ None of the above were true (to my knowledge) 

□ I'm not sure 


C8. At the time you applied for EAPSI, how many of the following had you authored/co- 
authored, edited/co-edited, developed/co-developed? 

Include works "in press" but do not include works "under review" or "in preparation." 
Include works published in electronic or printed format but do not count the same work 
more than once if it is available in multiple formats. 


Total number 
published/ 
in-press 

Number completed 
in collaboration with 
a foreign colleague 




Peer-reviewed journal articles 



Peer-reviewed conference publications (e.g. 
abstracts, conference papers, posters) 



Patents, registered or pending 



Book chapter(s) (e.g., in edited volumes) 


C9. By the time you applied for an EAPSI Fellowship, had you received a nationally competitive 
fellowship(s) to support your graduate studies? Do not include support you received 
directly from your graduate institution or support from a faculty member's grant funding. 

Nationally-competitive fellowships are unrestricted fellowships granted by a federal 
agency, private foundation, or similar organization directly to an individual graduate 
student (or graduate school applicant) for use at any graduate institution of his/her 
choosing. Check one only. 


□ Yes 

□ No 


MODULE D: PROFESSIONAL HISTORY 

In this section, we ask about various professional experiences and accomplishments that have 
occurred between [EAPSI_year] and a standard "reference date." We use a standard date so 
that all survey participants think about the same period of time when answering these 
questions. 

Dl. As of October 1, 2010, what is the highest degree you have completed ? Check one only: 

□ Bachelor's degree (BS, BA) 

□ Master's degree (MA, MS, MBA, etc.) 

□ Doctoral degree (PhD, EdD, MD, joint MD/PhD, JD, PsyD, ScD, etc.) 

□ Other degree- specify: [textbox, 50] 

If respondent holds Doctoral Degree, then go to Dlb. Else skip to Die. 

Dlb. [all respondents]: Since receiving your first doctoral degree, how many postdoctoral 
appointments have you held at institution(s) outside the United States? Please include 
any postdocs you held through October 1, 2010. Check one only. 

□ 0 
□ 1 
□ 2 

□ 3 or more 

Die. During the week of October 1, 2010 were you working for pay or profit? Work includes 
being self-employed, on a postdoctoral appointment, or on any type of paid or unpaid 
leave, including vacation. Check one only. 

□ i Yes, in a postdoctoral position -> GOTO D4 

□ 2 Yes, in another type of position (not a postdoctoral appointment) -> GOTO D2 

□ 3 No, I was still in graduate school then GOTO D6 

□ 4 No, I was not in graduate school and not working then -> GOTO D5 

IF Die = 3 (I was still in graduate school) then LET "CurrGrad = 1" (see note after Item D6) 


D2. Was your primary employer during the week of [October 1, 2010] an educational 
institution? Check one only 

□ Yes GOTO D2a 

□ No GOTO D3 


D2a. Was the educational institution where you worked a . . . Check one only 

□ Preschool, elementary, middle, or secondary school or system -> GOTO D4 

□ Two-year college, community college or technical institute 

□ Four-year college or university other than a medical school 

□ Medical school (including university-affiliated hospital or medical center) 

□ University-affiliated research institute 

□ Other educational institution [please specify]: [textbox, 150] 

*Four-year college/university includes doctoral-granting and non-doctoral-granting 
institutions. 

D2b. During the week of October 1, 2010 what type of academic position did you hold at 
this institution? Mark Yes or No for each item. 


Yes 

No 


□ 

□ 

President, Provost, or Chancellor (any level) 

□ 

□ 

Dean (any level), department head or chair 

□ 

□ 

Research faculty, scientist, associate or fellow 

□ 

□ 

Teaching faculty 

□ 

□ 

Adjunct faculty 

□ 


Other (please specify: [textbox, 75]) 


D2c. During the week of [October 1, 2010], what was your faculty rank? Check one only 

□ Not applicable: no ranks designated at this institution 

□ Not applicable: no ranks designated for my position 

□ Professor/Full Professor 

□ Associate Professor 

□ Assistant Professor 

□ Instructor 

□ Lecturer 

□ Other: [textbox, 75] 

D2d. What was your tenure status? Check one only 

□ Not applicable: no tenure system at this institution 

□ Not applicable: no tenure system for my position 

□ Tenured 

□ On tenure track but not tenured 

□ Not on tenure track 

Go to D4 


D3 only for respondents where D2 = NO 

D3. Which of the following best describes your primary employer during the week of October 
1, 2010? Check one only. 

□ SELF-EMPLOYED or a BUSINESS OWNER (non-incorporated or incorporated business, 
professional practice, or farm) 

□ PRIVATE SECTOR (for-profit or non-profit, including tax-exempt and charitable 
organizations; includes private colleges/universities 

□ Local GOVERNMENT (city, county, school district) 

□ State GOVERNMENT (including state colleges/universities) 

□ U.S. MILITARY service, activity duty or Commissioned Corps (e.g., USPHS, NOAA) 

□ U.S. federal GOVERNMENT 

□ OTHER type of employer: Please specify: [textbox, 300] 


D4. In performing the primary job you held during the week of [October 1, 2010], did you work 
with individuals located in countries other than the US? Check one only. If you held a 
postdoctoral appointment at this time, please consider that your primary job. 

□ Yes GOTO D4a 

□ No -»GOTOD5 

D4a. Did your work with individuals in countries other than the US involve . . . 

Mark Yes or No for each item. 


Yes 

No 


□ 

□ 

Sharing data or information? 

□ 

□ 

Sharing materials, equipment, or facilities? 

□ 

□ 

Preparing a joint publication? 

□ 

□ 

Jointly developing a product, process, or program? 

□ 

□ 

Collaborating on a research project? 

□ 


Other type of work? Specify: [textbox, 300] 


D4b. For the primary job you held during the week of October 1, 2010 were you 

employed at a location outside the U.S.? If you were based in the United States but 
travelled internationally for this job, you should answer "No." 

□ Yes 

□ No 


D5. Between [EAPSI_year] and October 1, 2010, did you receive any grants (as Principal 
Investigator or co-Principal Investigator), prestigious awards or honors based on your 
research? If you were an EAPSI fellow, do not count the EAPSI Fellowship itself. 

Check one response per row. 


Not applicable for my 
position (e.g., not 
eligible for 
grants/awards) 

Yes 

No 


□ 

□ 

□ 

Grant(s) as Principal Investigator 

□ 

□ 

□ 

Grant(s) as co-Principal Investigator 

□ 

□ 

□ 

Prestigious award(s)/honor(s) 


If any row in D5 = Yes: -> Former EAPSI Fellows, GO TO D5a; Unfunded applicants, SKIP 
to D5b 

If ALL rows in D5 = No or Not applicable -> GO TO D6. 

D5a. [former EAPSI Fellows only] Were any of these grants or awards based on research 
conducted or advanced during your EAPSI fellowship? Check one only 

□ Yes 

□ No 

□ Not sure 

D5b. What was the name of the most prestigious grant, award or honor for research you 
have received and who did it come from? 

(i) Name of award: [textbox, 300] 

(ii) The award was from: Check one: 

□ An institution where I was employed or my graduate institution 

□ A US-based professional association 

□ An international organization 

□ A U.S. government agency 

□ A private foundation in the U.S. 

□ None of the above 

(iii) Please type the full name of the awarding agency or organization: [textbox, 
300] 


D6. Between [EAPSI_year] and [October 1, 2010], how many of the following works had you 
published or produced (on your own or with others)? 

Include works "in press" but do not include works "under review" or "in preparation." 
Include works published in electronic or printed format but do not count the same work 
more than once if it is available in multiple formats. 

Make your best approximation if you do not know the exact number. 

The first row should not appear on-screen, and the column "EAPSI Fellows only" should 
appear only for FELLOWS not for UNFUNDED APPLICANTS 


Do not display this row on-screen 

All 

Respondents 

All respondents 

EAPSI Fellows only: 


Total 
(if none, 
enter "0") 

How many of these 
in collaboration 
with foreign 
colleague(s)? 

How many of these 
in collaboration 
with your EAPSI 
host? 

Peer-reviewed journal articles 




Peer-reviewed conference 
publications (e.g. abstracts, 
conference papers, posters) 




Patents, registered or pending 




Book chapter(s) (e.g., in edited 
volumes) 





AFTER ITEM D6, IF CurrGrad = 1 THEN GO TO Gl; ELSE IF CurrGrad = 0 THEN GO TO 
D7 


D7. Between [year of EAPSI] and [October 1, 2010], have you ever worked in a country other 
than the United States? ("Work" refers here to employment for pay or profit.) Do not 
include your EAPSI research fellowship here. 

□ Yes GOTO D7a 

□ No GOTO D8. 

D7a. For how many years (if less than 1 year, how many months) did you work in 
another country (or countries)? 

I worked in another country/countries for: Check one only. 

□ Less than 1 year ->Goto D7a_(i) 

□ 1 year or longer ->Goto D7a_(ii) 

D7a_(i). Enter number of months total: [textbox, 2] -> Go to D8 
D7a_(ii). Enter number of years total: [textbox, 2] Go to D8 
D7a(i) and D7a(ii) should appear on - screen together with D7a. 


D8. Between [EAPSI_year] and the week of [October 1, 2010], did you mentor any individuals 
from the United States who conducted research in another country?? Check all that apply 

□ I mentored undergraduate students who conducted research abroad 

□ I mentored graduate students who conducted research abroad 

□ I mentored postdocs who conducted research abroad 

□ I mentored faculty who conducted research abroad 

□ I mentored research scientists who conducted research abroad 

□ I mentored other individuals who conducted research abroad 

□ I did not mentor any individuals who conducted research abroad 


Former Fellows: GotoD9 
Unfunded applicants: Go to DIO 


D9 for Former EAPSI fellows only 

In this next question, we're interested in any effects that your participation in EAPSI may have 
had on people or institutions you've worked with in the U.S. since the end of your EAPSI 
fellowship. 

D9. Since the end of your EAPSI fellowship, did you do any of the following? Check one 
response per row 



Yes 

No 

Not 

applicable 

1 have taught my colleagues, students, or peers research 
methods that 1 learned during my EAPSI fellowship 

□ 

□ 

□ 

1 have shared with my colleagues resources (e.g., data, 
samples, materials) or tools (e.g., algorithms, software, 
instruments) that 1 collected developed during my EAPSI 
fellowship 

□ 

□ 

□ 


Former EAPSI fellows continue to DIO 


DIO. Between [EAPSI_year] and [October 1, 2010], did you do any of the following? Check 
one response per row 



Yes 

No 

Not 

applicable 

1 have established a program to foster international 
collaborations 

□ 

□ 

□ 

1 have hosted researchers or professional colleagues from 
another country at my institution 

□ 

□ 

□ 

1 have led a delegation of colleagues to visit a research 
laboratory, university, or business in another country 

□ 

□ 

□ 

1 have established or served in a leadership role for an 
international association for professionals in my line of 
work 

□ 

□ 

□ 


Fellows GO TO El 
Unfunded Applicants GO TO G1 


MODULE E: THE EAPSI FELLOWSHIP EXPERIENCE 


El. In preparation for your EAPSI fellowship, did you study a language spoken in your host site? 

Check all that apply. 

□ No, I was already familiar with the language spoken in the host site 

□ No, I did not study a language spoken in the host site 

□ Yes, I did some self-guided language study (i.e., individual study using books or 
computer-based instructional software, such as Rosetta Stone) 

□ Yes, I studied with a conversation partner or a tutor who was familiar with a language 
spoken in the host site 

□ Yes, I enrolled in a formal language training course led by an instructor (either an online 
course or "live" course) 


E2. During your EAPSI fellowship, did you attend or participate in any of the following types of 
cultural or leisure activities? Check all that apply 

□ Sightseeing 

□ Museums 

□ Festivals, holiday or religious ceremonies 

□ Outdoor activities to explore the landscape, geography 

□ Sporting events 

□ Non-scientific lectures or presentations 

□ Other (please specify): 

□ I did not participate in any cultural activities 

E3. During your EAPSI fellowship, did you attend or participate in any of the following types of 
professional activities? Check all that apply. 

□ Visit(s) to educational or research institutions other than my host institution 

□ Visit(s) to businesses/industrial laboratories 

□ Language courses or language study 

□ Lectures, colloquia, seminars in my field 

□ I gave a talk or presentation to researchers in my host site 

□ Networking with colleagues from institutions other than my host institution 

□ Other (please specify): 

□ I did not attend or participate in any professional activities 

E3a. Please describe one of the most memorable activities or events you experienced in 
your host site: 

[textbox, 2500] 



E4. Who was primarily involved in the following activities related to the EAPSI project(s) on 
which you worked? Mark one answer in each row. 



Mostly me 
independently 

Mostly the 
host or 
members of 
his/her group 
without me 

Me and the 
host or host's 
research group 
together 
about equally 

Not 

applicable 

Developing the ideas, hypotheses, broad 
framework, or vision for the research 
project 

□ 

□ 

□ 

□ 

Researching literature or research base 
relevant to the project 

□ 

□ 

□ 

□ 

Keeping records, tracking supplies, 
resources 

□ 

□ 

□ 

□ 

Developing instrumentation, software, 
equipment, or data collection processes 

□ 

□ 

□ 

□ 

Collecting data or carrying out simulations 

□ 

□ 

□ 

□ 

Analyzing data or observations 

□ 

□ 

□ 

□ 

Interpreting results 

□ 

□ 

□ 

□ 

Planning or developing follow-up work 
based on results 

□ 

□ 

□ 

□ 

Written, oral dissemination of results 
(publications, presentations) 

□ 

□ 

□ 

□ 





E5. Did you experience any of the following difficulties during your fellowship? Check all that 
apply. 

□ Inadequate access to space, facilities, equipment, computers, resources/supplies 

□ My role on the project was less than that merited by my skills/knowledge 

□ Not enough guidance from host/host's research group 

□ I was asked to do work that was someone else's responsibility 

□ Not given credit for my contributions to advancing a project 

□ Communication or language difficulties 

□ Logistical difficulties (e.g., with transportation, navigating bureaucracy, etc. ) 

□ Legal or medical difficulties in my host site 

□ I felt that my ideas were not treated with respect 

□ Encountered barriers or discomfort based on my gender 

□ Encountered barriers or discomfort based on my race/ethnicity 

□ Encountered barriers or discomfort based on my cultural or religious background 

□ Encountered barriers or discomfort based on a disability 

□ Other (please specify): [textbox, 300] 

□ None 

E6. During your EAPSI fellowship, who provided direct supervision while you were 
conducting your work? Check all that apply. 

□ The host researcher him/herself 

□ Another staff scientist 

□ A junior faculty member or post-doctoral fellow 

□ Other graduate student(s) 

□ A laboratory technician or other employee/worker 

□ No one, I was not supervised by anyone 

E7. Were any researchers from your graduate institution involved in the project(s) you 
conducted during EAPSI? Check all that apply. 

□ My graduate advisor 

□ Other faculty at my U.S. graduate institution 

□ Other graduate students at my U.S. graduate institution 

□ Undergraduate students at my U.S. graduate institution 

□ None of the above 


E8. Please indicate how satisfied you were with various aspects of your EAPSI experience. Check 


one answer in each row. 


Accommodations & Logistics 

Very 

dissatisfied 

Somewhat 

dissatisfied 

Somewhat 

satisfied 

Very 

satisfied 

Lodging 

□ 

□ 

□ 

□ 

Fellowship support amount 

□ 

□ 

□ 

□ 

Fellowship duration 

□ 

□ 

□ 

□ 

Research/laboratory facilities at the 
host institution 

□ 

□ 

□ 

□ 

Access to the internet 

□ 

□ 

□ 

□ 

Your host 

Very 

dissatisfied 

Somewhat 

dissatisfied 

Somewhat 

satisfied 

Very 

satisfied 

Frequency of meetings with host 

□ 

□ 

□ 

□ 

Match between host's and my 
research interests 

□ 

□ 

□ 

□ 

Host's expertise in his/her field 

□ 

□ 

□ 

□ 

The level of the host's intellectual 
contribution to our joint research 
project 

□ 

□ 

□ 

□ 

Host's efforts to help me meet other 
researchers 

□ 

□ 

□ 

□ 

Host's inclusion of me in research 
group/laboratory, meaningful 
collaboration 

□ 

□ 

□ 

□ 

Guidance or mentoring provided by 
host 

□ 

□ 

□ 

?? 

Your experiences 

Very 

dissatisfied 

Somewhat 

dissatisfied 

Somewhat 

satisfied 

Very 

satisfied 

Opportunities to experience and 
learn about culture, history, 
geography, etc. in my host site 

□ 

□ 

□ 

□ 

The quality of research 1 was able to 
conduct 

□ 

□ 

□ 

□ 

Match between scope or goals of 
project and duration of EAPSI 

□ 

□ 

□ 

□ 

Professional connections made 
during the fellowship 

□ 

□ 

□ 

□ 


E8a. If you would like to elaborate or comment further on areas of satisfaction or 
dissatisfaction with your EAPSI experience, please do so. 
textbox, 2500 



MODULE F: AFTER YOUR EAPSI FELLOWSHIP 


FI. Since the conclusion of your EAPSI fellowship and [October 1, 2010], have you collaborated 
or communicated with your host scientist? Check one only 

□ I have collaborated on a research project -> GOTO Fla 

□ I have communicated with my host but haven't collaborated further on research 

GOTO Flc 

□ I have not communicated with my host GOTO Flc 


Fla. What was the extent of your collaboration? Check all that apply 

□ I have a position in the host's group 

□ I have a position at the host's institution 

□ We exchanged ideas, data, ideas, research results, or tools 

□ We co-authored papers 

□ We co-advised students 

□ We visited each other at our institutions 

□ Other - specify [textbox, 300] 


Fib. How recently has the latest collaboration with your host occurred? Check one only 

□ Within the past 6 months GOTO F2 

□ Within the past 12 months GOTO F2 

□ 1-2 years ago GOTO F2 

□ 3 or more years ago GOTO F2 


Flc. Why do you no longer collaborate with your former host? Check all that apply 

□ Our research interests diverged 

□ One or both of us lacked funding needed to maintain collaboration 

□ Language differences have hindered further collaboration 

□ Political or cultural differences have hindered further collaboration 

□ Geographic distance has hindered further collaboration 

□ I did not think that further collaboration would be beneficial for me 

□ My host did not actively pursue or maintain further collaboration with me 

□ My colleagues discouraged me from continuing this collaboration 

□ My advisor discouraged me from continuing this collaboration 

□ One (or both) of us is too busy with other projects 

□ Other - specify [textbox, 300] 


F2. Did participating in EAPSI make you qualified for a broader range of opportunities after the 
fellowship ended? Check one only. 

□ Yes, EAPSI did broaden my opportunities-^ GOTO F2a 

□ No, EAPSI did not broaden my opportunities -> GOTO F2b 

□ I am not sure -> GOTOF3 

F2a. Describe how EAPSI broadened your opportunities: 

[textbox, 2500] 


F2b. Did EAPSI constrain your opportunities? Check one only. 

□ Yes -> Go to F2c 

□ No Go to F3 

F2c. Describe how EAPSI constrained your opportunities: 
textbox, 2500 


F3. Which of the following professional benefits occurred as a result of your participation in 
EAPSI? Check all that apply 

□ My work at the host institution resulted in a substantial advancement in my research 

□ My work at the host institution opened up new areas of investigation 

□ I became familiar with the scientific enterprise in my EAPSI site 

□ I became committed to international research collaboration 

□ I made valuable connections to researchers outside the U.S. 

□ My career goals changed from an academic career to a non-academic career 

□ My career goals changed from a non-academic to an academic career 

□ I decided to pursue a graduate degree in a different discipline than the one I was 
pursuing when I began my EAPSI fellowship 

□ EAPSI participation made me more competitive for jobs I was interested in 

□ EAPSI participation made me consider professional opportunities I would not have 
considered in the past 

□ None of the above 




F4. Which of the following additional benefits occurred as a result of your participation in 
EAPSI? Check all that apply 

□ Research methods or ideas that I learned benefited others in my institution 

□ Samples that I collected or tools that I developed benefited others in my institution 

□ My peers became interested in international collaboration 

□ Others in my research group (in the U.S.) began an international research collaboration 

□ Researchers that I met during my fellowship joined my research group 

□ Other - specify [textbox] 

□ None of the above 

F5. Which of the following personal benefits occurred as a result of your participation? 

Check all that apply. 

□ I became comfortable with the traditions and culture of my host site 

□ I made personal connections in the places I visited during the fellowship 

□ I gained proficiency in another language 

□ I decided to live outside the United States (i.e. at least 6 months) 

□ None of the above 

F6. Which of the following are true? Check Yes or No for each: 


Yes 

No 


□ 

□ 

Completion of my graduate degree was delayed because of my 
participation in EAPSI 

□ 

□ 

Competition was introduced between my research group and the host 
because of my participation in EAPSI 

□ 

□ 

1 lost an important career or educational opportunity by participating in 
EAPSI 

□ 

□ 

1 am more skeptical about international collaboration than before EAPSI 


F7. What was the most positive aspect of your EAPSI experience? [textbox, 2500] 


F8. What was the most challenging aspect of your EAPSI experience? [textbox, 2500] 


F9. Would you recommend the EAPSI fellowship to another graduate student seeking an 
international experience? Check one only. 

□ Yes 

□ No 

F9a. Why or why not? [textbox, 2500] F9a should appear on same screen as Item F9 

F10. Would you recommend your EAPSI host scientist to another graduate student seeking an 
international experience? Check one only. 

□ Yes 

□ No 

Fll. Your fellowship took place in [EAPSI_actual_site], Would you recommend this EAPSI site to 
another graduate student seeking an international experience? Check one only. 

□ Yes 

□ No 

F12. What would you change about the program? [textbox, 2500] 


F13. What was the most important contribution of EAPSI participation to your career? 
[textbox, 2500] 


MODULE G: DEMOGRAPHIC INFORMATION 


Gl. What is your gender? 

□ Male 

□ Female 

G2. What is your ethnicity? Check one only. 

□ Hispanic or Latino 

□ Not Hispanic or Latino 

G3. What is your race? Check one or more. 

□ American Indian or Alaska native 

□ Asian 

□ Black or African American 

□ Native Hawaiian or other Pacific Islander 

□ White 

G4. What is your citizenship status? Check one only. 

□ United States citizen since birth -> Go to G5 

□ United States citizen, naturalized -> Go to G4a 

□ Non-U. S. citizen with a permanent 

U.S. Resident Visa ("Green Card") -> Go to G4a 

□ Non-U. S. citizen with a temporary U.S. Visa Go to G4a 

G4a. How old were you when you began living in the United States? (Enter age in years): 
[textbox, 2] 


The next question is designed to help us better understand the career paths of individuals 
with different physical disabilities. 

G5. What is the USUAL degree of difficulty you have Unable 


with: Mark one answer for each item. None Slight Moderate Severe to do 


1 SEEING words or letters in ordinary newsprint (with 
glasses/contact lenses, if you usually wear them) 

□i 

□2 

□3 

□4 

□5 

2 HEARING what is normally said in conversation with 
another person (with hearing aid, if you usually wear 
one) 

□ i 

□2 

□3 

□4 

□5 

3 WALKING without human or mechanical assistance or 
using stairs 

□i 

□2 

□3 

□4 

□5 

4 LIFTING or carrying something as heavy as 10 pounds, 
such as a bag of groceries 

□i 

□2 

□3 

□4 

□5 


If all 4 rows in G5 are marked "None," Go to Thank you screen. Else display Item G5a underneath Item 
G5. 

G5a. What is the earliest age at which you first began experiencing any difficulties in any of 


these areas? 


□ Since birth; or Enter age in years: _ 
Programmer note: age in years valid range = 0 to 99 


SUBMIT SURVEY 



Thank you screen: After hitting SUBMIT, respondent sees this screen. 


EAPSI applicant survey 


Conclusion 

Thank you for completing this survey. If you have any general comments about the survey, 
please write them below. 

Textbox, 2500 


If you have any questions about this survey or the study, please contact Dr. Alina Martinez, 

Study Director, Abt Associates, at (866) 421-6223 (toll-free within the US) or email her at 
EAPSI_survey@abtassoc.com. You may also contact John Tsapogas, who is overseeing this study 
at NSF, with any questions or comments: itsapoga@nsf.gov . 

Thank you for your assistance. We greatly appreciate your time and consideration. 





CONFIRM FELLOWSHIP SCREEN, SECTION R 

NSF's records may be in error. However, before continuing, we'd like to note that some 
summer institutes that are part of the East Asia and Pacific Summer Institutes (EAPSI) program 
are also known by different names. For example, all of the following summer institutes are part 
of the EAPSI program: 

■ The Summer Program in Japan, a joint program of the Japan Society for the Promotion of 
Science (JSPS) and NSF; 

■ The Korea Summer Institute, a joint program of the Korea Science and Engineering 
Foundation (KOSEF) and NSF; 

■ The Summer Institute in Taiwan (SIT), a joint program of Taiwan's National Science Council 
(NSC) and NSF; 

■ The EAPSI summer program in Australia, supported by the Australian Academy of Science 
and NSF; 

■ The China Summer Institute, supported by NSF and the Chinese Ministry of Science and 
Technology (MOST), the Chinese Academy of Sciences (CAS), and National Natural Science 
Foundation of China (NSFC). 

■ The EAPSI summer program in New Zealand, coordinated by the Royal Society of New 
Zealand and supported by the New Zealand Ministry of Research, Science and Technology 
(MoRST) and NSF; 

■ The EAPSI summer program in Singapore, co-sponsored by Singapore's National Research 
Foundation and NSF. 

If AWDSTATUS = 1 than display K1 
If AWDSTATUS = 2 then display K2 


Programmer note: the above list of programs must appear on same webpage/screen as K1 
Kl. If you participated in any of the above, these are each part of the EAPSI program. 

Did you participate in any of the above? 

□ Yes, I did -} if respondent came here via Intro screen, return to that 
intro screen. If respondent came here from A2.1 go to A2.3a below 

□ No, I applied but I did not participate in the EAPSI program -} Go TO S2 

□ No, I did not participate; nor did I ever applied to any of the above 
programs-} Go TO EXIT SCREEN 

□ I'm not sure -» Go TO EXIT SCREEN 

A2.3a Did you participate in this EAPSI program in [EAPSI_Year]? If you participated in an 
EAPSI program more than once, we are asking about your most recent EAPSI fellowship 

□ Yes 

□ No, it began in (Enter four-digit year): [yyyy] Set EAPSI_Year = entry and FLAG 
EAPSI_YEAR_CHANGE = 1 

AFTER A2.3a, GO TO A3 


Programmer note: the above list of programs must appear on same webpage/screen as K2. Also note 
that the ONLY way a respondent arrives at K2 is via the intro screen (not via A2.2 - A2.2 directs the 
respondent to Section S). 

K2. If you submitted an application for any of the above, these are each part of the EAPSI 
program. 

Did you apply for any of the above? 

□ Yes, I did -} Return to intro screen. 

□ No, I did not -} Go TO EXIT SCREEN 

□ I'm not sure -} Go TO EXIT SCREEN 


SECTION S 

For this study, we are seeking former EAPSI applicants who prepared and submitted an 
individual research proposal and application to the National Science Foundation for its EAPSI 
program. Some students may have applied to the EAPSI program but travelled to Australia, 
China, Japan, Korea, New Zealand, Taiwan, or Singapore under the sponsorship of a U.S. faculty 
member who accompanied students as part of a different NSF-funded program. 

Did your summer research program in Australia, China, Japan, Korea, New Zealand, Taiwan, or 
Singapore result from an application that you prepared and submitted to NSF, or did you go to 
one of these places under a U.S. faculty member's sponsorship? 

□ My EAPSI summer research fellowship was the result of an application I prepared and 
submitted -> go to SI 

□ I submitted an application myself, but the international summer research program in 
which I participated was under the sponsorship of a U.S. faculty member who 
accompanied students abroad go to EXIT SCREEN 

□ I'm not sure go to EXIT SCREEN 

SI: Please accept our apology for the mistake in our records. We especially need you to 
complete this survey. This survey focuses on your international experiences, professional 
achievements, the type of work you do currently and the types of collaborations you may have 
with scientists and engineers in the US and other countries. If you would please continue, we 
have updated your status as a former EAPSI summer research fellow. You indicated that you 
participated in EAPSI in [EAPSI_year] in [EAPSI_actual_site], SET AWDSTATUS = 1, SET 
FLAG_AWDSTATUS_CHANGE =1 and Go to A3 when respondent clicks on Continue 

CONTINUE Go to A3 


S2: Please accept our apology for the mistake in our records. We especially need you to 
complete this survey. This survey focuses on your professional achievements, the type of work 
you do currently and the types of collaborations you may have with scientists and engineers in 

the US and other countries. The value of this study depends on the participation of individuals 
who applied for one of the above EAPSI programs, even if you did not participate. Please 
click here to continue with to the survey. CONTINUE 

SET AWDSTATUS = 2 AND GOTO Item A2.3b. FLAG RESPONDENT AS 
'FLAG_AWDSTATUS_CHANGE=1' for our analysis purposes. 

A2.3b Did you apply for a [EAPSI_ Year] EAPSI summer research fellowship? 

□ Yes 

□ No, I applied for a different year (Enter four-digit year): [yyyy] Set EAPSI_ Year = 
entered text and FLAG EAPSI_YEAR_CHANGE = 1 

AFTER A2.3b, GO TO A3 


EXIT SCREEN screen: 

If we have identified the wrong respondent or have erroneous information, display this screen 
and FLAG this respondent for Abt follow-up (weekly basis): 


EAPSI applicant survey 


Please accept our apology. 

The information you supplied suggests that you are not eligible to participate in this study or 
that we have reached you in error. We regret any inconvenience to you. If you have any 
questions about this study or you would like to make a comment, please contact one of the 
following individuals: 

■ Dr. Alina Martinez, Study Director at Abt Associates: EAPSI_survey@abtassoc.com, or 
(866) 421-6223 (toll free within the U.S.); 

■ John Tsapogas, Office of International Science and Engineering, NSF: jtsapoga@nsf.gov. 

May we have permission to contact you by telephone to clarify your responses here? Entering 
your number does not obligate you to answer any questions. 

□ Yes, you may reach me at . [telephone digit entry] Go to Best Times. 

□ No, please do not contact me. 

Best time(s) to call ( check all that apply): 

□ Weekdays (9 to 5pm) 

□ Evenings (5 to 8pm) 

□ Saturday or Sunday 9-5pm 

DO NOT REQUIRE RESPONSES ON THIS EXIT SCREEN. NO REMINDER ON THIS SCREEN IF IT IS LEFT 
BLANK. 

Exit screen must have a PREVIOUS ITEM button to allow for a respondent who accidentally marked a 
response not intended. 




DEFAULTS for SKIP PATTERNS IF NO RESPONSE 
REMINDERS FIRST TIME NO RESPONSE 

• No items require a response. 

• No items require a reminder to enter an answer/select a response unless indicated below. 
If a reminder is needed for the first time user leaves an item blank, re-display the item 
with the text below shown on-screen in bold, red text. 

DEFAULTS for SKIP PATTERNS 

Module A: If Al, A2.1/A2.2, and/or A4a. are blank, no reminder needed but use the 
default below for skip patterns: 

Al. Based on the information that you provided to NSF in your application, your name is [First 
Name, Middle Name/initial, Last Name], Is this correct? 

□ Yes, this is correct. 

FELLOWS GOTO A2.1 

UNFUNDED APPLICANTS GOTO A2.2 

[Former Fellows only]: 

A2.1 Based on the information contained in the NSF EAPSI database, you participated in the 
National Science Foundation's East Asia and Pacific Summer Institutes program (EAPSI) in 
the summer of [EAPSI_year], Is this correct? 

If you participated in EAPSI more than once, we refer here, and throughout the rest of the 
survey, to your most recent EAPSI experience unless specifically stated otherwise. 

□ Yes, this is correct. GOTO A3 
[Unfunded applicants only] 

A2.2. Based on the information contained in the NSF database, you most recently applied for 
EAPSI in [year], but did not participate in a fellowship. Is this correct? 

□ Yes, this is correct. GOTO A3 


A4. Based on the information that you provided to NSF in your application, you were enrolled at 
[University] when you most recently applied for an EAPSI fellowship. Is this correct? 

□ Yes 


A5a. As of [October 1 st , 2010], had you earned this degree? 
□ Yes Go to A5b 


REMINDERS to DISPLAY on FIRST NO RESPONSE 

A3. Based on the information contained in the NSF database, when you most recently applied 
for an EAPSI fellowship, the first choice of host site you listed was [EAPSI_app_site]. (If you 
received an EAPSI fellowship, you may have visited a different site, but here we want to 
know what site you listed in your application as your first choice.) 


Is this correct? 


Please select a response before continuing to the next item, [it is ok if this item is still 
left blank] 

A3a. Some EAPSI fellows did not receive their first choice of host site. In which site was your 
EAPSI fellowship? 

No response selected. If you do not recall, you may leave this item blank and go to the 
next item. 

A5. What graduate degree were you pursuing when you most recently applied for the summer 
EAPSI fellowship? 

No response selected. If you do not recall, you may leave this item blank and go to the 
next item. 

A5a. As of [October 1 st , 2010], had you earned this degree? 

No response selected. If you do not recall, you may leave this item blank and go to the 
next item. 

A7. What discipline were you studying as a graduate student when you applied for an EAPSI 
fellowship? Please select the field that best matches your discipline. First, indicate 
which one of four broader areas in which your discipline falls: Check one only: 

Please select a response before continuing to the next item, [it is ok if this item is still 
left blank] 

□ i Sciences 

(e.g., Biological/Life, Chemical, Computer/Information, Environmental, 
Earth/Atmospheric/Oceanographic, Geosciences, Mathematical/Statistical, 
Physics, Psychology) 

□ 2 Social Sciences 

(e.g., Economics, Sociology, Anthropology/Archaeology, Political Science, 
Geography, Linguistics, other) 

□ 3 Health/Medical fields 

□ 4 Engineering 

[and it's also ok if no 3-digit code is selected within "Sciences" or "Social Sciences" or 
"Health/Medical" or "Engineering" -- no reminder needed] 


Cl. Using a 4-point scale, what was your overall undergraduate grade point average (GPA)? 

If you have more than one bachelor's degree, give your overall grade point average for your 
first bachelor's degree. Check one only. 

No response selected. If you're not sure or do not recall, please make your best estimate, 
[it is ok if this item is still left blank] 

Dl. As of October 1, 2010, what is the highest degree you have completed ? Check one only: 

Please select a response before continuing to the next item, [it is ok if this item is still left 
blank] 


Die. During the week of October 1, 2010 were you working for pay or profit? Work includes 
being self-employed, on a postdoctoral appointment, or on any type of paid or unpaid 
leave, including vacation. Check one only. 

Please select a response before continuing to the next item, [it is ok if this item is still 
left blank] default = Yes, in another type of position (not a postdoc) 

D2. Was your primary employer during the week of [October 1, 2010] an educational 
institution? Check one only 

No response selected. If you did not mean to leave this item blank, please check one: 

[if left blank, default is "yes" and go to D2a] 

D4. In performing the primary job you held during the week of [October 1, 2010], did you work 
with individuals located in countries other than the US? Check one only. If you held a 
postdoctoral appointment at this time, please consider that your primary job. 

No response selected. If you did not mean to leave this item blank, please check one: 

[if left blank, default is "no" and go to D5] 


D7. Between [year of EAPSI] and [October 1, 2010], have you ever worked in a country other 
than the United States? ("Work" refers here to employment for pay or profit.) 

No response selected. If you did not mean to leave this item blank, please check one: 
[if left blank, default is "no" and go to D8] 

□ Yes GOTO D7a 

□ No GOTO D8. 


Survey of Graduate Advisors of 
former East Asia and Pacific Summer Institute (EAPSI) Fellows 
for the National Science Foundation 


Programming notes (were not displayed in the actual survey) appear in red or brown text throughout. 
Hyperlinked text appeared and wdtiflliiifted in this document but were not formatted this way 

onscreen unless noted. FAQs opened in a popup window. 

General guidelines: 

Each screen displayed one question at a time unless otherwise indicated. 

Each screen displayed a standard set of Navigation buttons shown below. 


EAPSI Advisor Survey 


Module A: Verifying Information about You 

We would like to ask you about your experiences as faculty advisor to a former graduate Fellow 
participant in the East Asia and Pacific Summer Institute (EAPSI) program. Please respond to 
the best of your recollection. To begin, we'd like to confirm that we've reached the appropriate 
person. 

Please note: We will not reveal responses that you provide on any question in this survey to any of the fellows 
that you advised. Neither will any data that can identify any specific fellow or graduate advisor be shared with 
the sponsor of this survey (NSF). 

Based on information in NSPs EAPSI database, you were the faculty advisor for Brian Sullivan, a former 
graduate student who participated in the East Asia and Pacific Summer Institute (EAPSI) program while he or 
she was in graduate school. Were you a faculty advisor to this individual? 

Mark one answer 

© Yes, I was a faculty advisor to this individual. 

© No, I have never been a faculty advisor to this individual. 

© No, I do not know this individual. I believe you have reached me by mistake. 




PREVIOUS ITEM 


NEXT ITEM 


Some items contained placeholders for fields that were programmed to be auto-filled from the sample 
file for each respondent. These fields are enclosed in [brackets] and printed in brown. Textboxes for 
open-text responses are indicated with [textbox, 150] where the number indicates the length of the 
field (how many characters the respondent may type) 


EAPSI ADVISOR SURVEY 


Page 1 





Sample variables: 


EAPSI_appSite: Name of the country or site that FELLOW NAME APPLIED TO. 

EAPSI_actualSite: Site where FELLOW NAME ACTUALLY PARTICIPATED -not really a sample 
variable, will be updated by advisor during survey (see Item A5 below) 

EAPSI_Dropdown Menu: 

Values: 

[blank] 

Australia 

China 

Korea 

Japan 

New Zealand 

Singapore 

Taiwan 


EAPSI_YEAR: the year that the respondent hosted the EAPSI fellow named in the survey. 
Values: 2000, 2001, 2002, 2003 2004, 2005, 2006, 2007, 2008, or 2009 

Host institution: the name of the university, research center, company where the host worked 
when EAPSI fellow visited. 

Grad_institution: the name of the U.S. graduate university where the Fellow was enrolled at 
the time of EAPSI 

Fellow Name: name of individual EAPSI fellow 


EAPSI ADVISOR SURVEY 


Page 2 



Survey of Graduate Advisors of 
former East Asia and Pacific Summer Institute (EAPSI) Fellows 
for the National Science Foundation 


Welcome and thank for your interest in this study. This survey is being conducted by Abt Associates Inc. and 
our subsidiary, AbtSRBI, for the National Science Foundation (NSF) to learn about the perspective of advisors to 
graduate student participants in NSF's East Asia and Pacific Summer Institute (EAPSI) program. This survey will 
give NSF information about the professional characteristics and international collaborations of U.S. scientists and 
engineers and help NSF improve programs intended to foster a globally engaged scientific and engineering 
workforce. You are receiving this survey because you are listed in the EAPSI database as the faculty advisor of a 
graduate student who was an EAPSI fellow. We estimate that it will take approximately 15 minutes to complete 
the survey. 

Confidentiality and Participation 

Participation in the survey is voluntary and nonparticipation will have no impact on you or your institution. You 
may skip questions on the survey or discontinue participation at any time. There are minimal risks associated 
with your participation. We take your privacy very seriously. Your responses to this survey will be protected 
under the Privacy Act. There is minimal risk of breach of confidentiality, and we have put in place procedures to 
minimize this risk. Reports will never identify you by name, and information from the study will only be reported 
in the aggregate at the program level, combined with about approximately 500 other responses. We will not 
reveal responses that you provide on any question in this survey to any of the fellows that you advised. Neither 
will any data that can identify any specific fellow or graduate advisor be shared with the sponsor of this survey 
(NSF) . When we receive your survey we will detach and store separately your name and other identifying 
information that could be used to link you to your survey responses. Survey responses will be stored on a secure 
drive that is only accessible to members on the study team. Only study team web technicians and data analysts 
from Abt Associates and AbtSRBI will see individual responses that can be linked to you. Survey data files will be 
shared with NSF at the end of the study, only after study team members have examined the data to be free of 
any information that could help identify you; this cleaning includes procedures to prevent someone from 
inferring your identity by analyzing non-identifying data. Flence, we encourage you to respond candidly about 
your experiences. Separate from your individual responses to the survey we will provide NSF any updated 
contact information we have found or requested from you. None of this contact information will be linked in 
any way to your survey responses. At the conclusion of the study, Abt Associates and AbtSRBI will destroy all 
records, electronic or otherwise, that link you to your survey responses. 

Questions 

If you have questions about the study, please contact the study director, Alina Martinez of Abt Associates Inc. at 
(866) 421-6223 (toll free within the U.S.) or email her at EAPSI_survey@abtassoc.com. You may also contact the 
evaluation's program officer at NSF, John Tsapogas (jtsapoga@nsf.gov). If you have questions about your rights as 
a research participant, you may contact Teresa Doksum, the Abt Institutional Review Board Administrator at (877) 
520-6835 (toll free within the U.S.) or by email: irb@abtassoc.com. To learn more about this study, please refer to 
the Frequently Asked Questions page. 


Consent 

Please click BEGIN if you agree to participate in this study. 


BEGIN 


This study's IRB approval number is #0494, valid from 8/6/2010 to 8/5/2011. For questions, please contact 
Teresa Doksum, IRB Administrator, Abt Associates, at IRB@abtassoc.com. The valid OMB control no. for this 
information collection is 3145-0214 (Expires on 12/31/13) 
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Navigating through the survey: 

As you work through the survey, your responses are automatically saved. You may change a response 
by clicking on the PREVIOUS ITEM button. Use theNEXT ITEM button to advance to the next 
question. At any time, you may close the survey and resume where you left off at a later time. On 
each page of the survey, anEAQs button is provided if you have a question during the survey or need 
information about how to contact the survey administrator. 

When you have completed the survey, please click on theSUBMIT button at the end of the survey. 
You may submit the survey even if there are some questions that you choose not to answer. 


CONTINUE 
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FREQUENTLY ASKED QUESTIONS 


• What is the East Asia and Pacific Summer Institutes (EAPSI) Program? 

• Why are you doing this study? 

• Why have I been selected to participate in this survey? 

• How did you get my contact information? 

• How long will this survey take to complete? 

• How will you use my comments? 

• Does this study have human subjects review clearance? 

• Who is funding the study? 

• Who are Abt Associates Inc and AbtSRBI? 


What is the East Asia and Pacific Summer Institute (EAPSI) Program? 

The East Asia and Pacific Summer Institute (EAPSI) Program is a National Science 
Foundation (NSF) program that provides funding for U.S. graduate students to spend eight to 
ten weeks collaborating with a host researcher in one of seven locations in the East Asia and 
Pacific region: Australia, China, Japan, Korea, New Zealand, Singapore, and Taiwan. 

Why are you doing this study? 

NSF is interested in learning about the characteristics of current and former graduate students 
who participated in the EAPSI program between 1999 and 2009. NSF is particularly interested 
in how the graduate advisors of EAPSI Fellows’ view the program and its benefits (or costs) to 
graduate students’ subsequent educational and career paths. This information will help NSF 
understand the usefulness and relevance of international research training for scientists and 
engineers. 

Why have I been selected to participate in this survey? 

You have been selected to participate because we have identified you as having served as 
graduate advisor to a former EAPSI fellow. 

How did you get my contact information? 

We identified you from records maintained by the National Science Foundation on prior EAPSI 
Fellows’ applications. We then obtained your contact information through an internet search. 

How long will this survey take to complete? 

We estimate that the survey will take about 15 minutes. 

How will you use my comments? 

Responses from all survey respondents will be used to answer questions about the graduate 
training, international experiences, and career paths of former EAPSI Fellows. We will not 
reveal responses that you provide on any question in this survey to any of the fellows that you 
advised. Neither will any data that can identify any specific fellow or graduate advisor be shared 
with the sponsor of this survey (NSF) . 
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What happens if I don’t answer this survey? 

Participation in the survey is voluntary and nonparticipation will have no impact on you or your 
institution. Study participation will not affect your current NSF funding or future applications to 
NSF for funding/grants. 

Has this study received human subjects review clearance? 

Yes, the study was approved by Abt Associates’ Institutional Review Board. If you have any 
concerns about your participation in this survey, please contact Teresa Doksum, Institutional 
Review Board Administrator at Abt Associates, at (877) 520-6835 or via email at 
irb @ abtassoc .com. 

Who is funding the study? 

The study has been funded by the National Science Foundation under contract 
GS-10F-0086K. Abt Associates and AbtSRBI will complete the study. 

Who are Abt Associates Inc and AbtSRBI? 

Abt Associates is an independent research firm headquartered in Cambridge, Massachusetts. 
AbtSRBI is a wholly-owned subsidiary of Abt Associates specializing in large-scale data 
collection and public opinion research. NSF has contracted with Abt Associates and AbtSRBI to 
design and implement a survey of the graduate advisors of former participants in NSF’s EAPSI 
program. 
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EAPSI Study: Graduate Advisor Survey 


Module A: Verifying Information about You 

We would like to ask you about your experiences as faculty advisor to a former graduate 
Fellow participant in the East Asia and Pacific Summer Institute (EAPSI) program. Please 
respond to the best of your recollection. To begin, we'd like to confirm that we've reached 
the appropriate person. 

Please note: We will not reveal responses that you provide on any question in this survey to any 
of the fellows that you advised. Neither will any data that can identify any specific fellow or 
graduate advisor be shared with the sponsor of this survey (NSF). 


Ala. Based on information in NSF's EAPSI database, you were the faculty advisor for [Fellow 
name], a former graduate student who participated in the East Asia and Pacific Summer 
Institute (EAPSI) program while he or she was in graduate school. Were you a faculty 
advisor to this individual? Mark one answer 

□ Yes, I was a faculty advisor to this individual. 

□ No, I have never been a faculty advisor to this individual. EXIT SURVEY. 

□ No, I do not know this individual. I believe you have reached me by mistake. EXIT 
SURVEY 

Alb. Based on the information NSF's EAPSI database, your name is [First Name, Middle 
Name/Initial, Last Name], Is this information correct? Mark one answer 

□ Yes. 

□ No, my name has changed or is misspelled above. My name is: [Textbox, 75] 

□ No, I am not the person named above. I believe you have reached me by mistake. EXIT 
SURVEY 


If Gradjnstitution = <missing> then GO TO A2 

Else If Gradjnstitution = <not missing> then GO TO A2a. 

A2. At what graduate institution were you a faculty member in [EAPSI_YEAR] when you served 
as this student's graduate advisor? Please type in the full name of the university without 
abbreviations. If the university has more than one campus, please indicate the specific 
campus (e.g., University of California, Los Angeles) 

Name of graduate institution: [textbox, 300] Set Gradjnstitution = text entered 
here and GOTO A2a 


A2a. As of October 1, 2010, were you at [Gradjnstitution]? Mark one answer 
□i Yes 

□ 2 No, I was at a different institution on October 1, 2010 
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A3. What was your faculty rank (or other title) at the time you were a graduate advisor to 
[Fellow Name] (i.e., in [EAPSI_Year])? Check all that apply 

□ Professor Emeritus/a 

□ Professor/Full Professor/Named Professor/Distinguished Professor 

□ Associate Professor 

□ Assistant Professor 

□ Adjunct Professor 

□ Visiting Professor 

□ Department Chairperson 

□ Graduate Program Chairperson 

□ Dean (any level) 

□ Other rank or title (Please specify): [textbox, 150] 

A4. What is your current title/current titles? 

□ Unchanged since [EAPSI_Year] 

Changed: My current title is/titles are: Check all that apply 

□ Professor Emeritus/a 

□ Professor/Full Professor/Named Professor/Distinguished Professor 

□ Associate Professor 

□ Assistant Professor 

□ Adjunct Professor 

□ Visiting Professor 

□ Department Chairperson 

□ Graduate Program Chairperson 

□ Dean (any level) 

□ Other rank or title (Please specify): [textbox, 150] 


A5. Did your former graduate student ([Fellow Name]) participate in EAPSI in [EAPSI_appSite] or 
in another of the EAPSI sites? Mark one answer 

□ Yes, this student participated in EAPSI in [EAPSI_appSite] Set EAPSI_actualSite = 
EAPSI_appSite 

□ No, I believe this student participated in EAPSI in [EAPSI site dropdown menu] Set 
EAPSI_actualSite = selected value from menu unless selection is [blank]. If selection is 
blank PRESENT REMINDER: "Please click on the menu to select a location." If still 
[blank] then Set EAPSI_actualSite = EAPSI_appSite 

□ I do not recall, or I am not sure. Set EAPSI_actualSite = EAPSI_appSite 
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Module B: Your role in EAPSI Fellow's application to the program 


NSF's EAPSI program records indicate that [Fellow Name] participated in the EAPSI program 
during the summer of [EAPSI_YEAR]. Items on the rest of this survey ask about your 
involvement with the Fellow, the Fellow's EAPSI experience and your perceptions of the 
EAPSI program. Please answer to the best of your recollection. 

Bl. Before your student [Fellow Name] applied for an EAPSI fellowship, how familiar were you 
with the EAPSI program? Mark one answer 

□ Very familiar -> GO TO B2 

□ Somewhat familiar -> GO TO Bla 

□ Not very familiar -> GO TO Bla 

□ I had never heard of it -> GO TO Bla 

□ I cannot recall GO TO Bla 

Bla. Were you familiar any of the following programs? These programs were alternative names 
for the EAPSI institutes in earlier years. Check all that apply. 

□ The Summer Program in Japan 

□ The Korea Summer Institute 

□ The Summer Institute in Taiwan 

□ The Summer Institute in China 

□ The EAPSI program in Australia 

□ The EAPSI program in New Zealand 

□ The EAPSI program in Singapore 

□ None of the above 


B2. In [EAPSI_YEAR], did your institution do any of the following to publicize the EAPSI program 


to graduate students? Check one response in each row 



YES 

NO 

DO NOT RECALL 

Provided application information about EAPSI in 
departmental newsletters, emails, listservs 

□ 

□ 

□ 

Posted EAPSI information on departmental 
bulletin boards 

□ 

□ 

□ 

Maintained current EAPSI program information 
at the student research/internship/ or career 
center 

□ 

□ 

□ 

Held on-campus meetings about EAPSI 

□ 

□ 

□ 

Former EAPSI fellows presented their EAPSI 
experiences to other graduate students 

□ 

□ 

□ 

Actively encouraged individual graduate 
students to apply for the EAPSI program 

□ 

□ 

□ 

Other. Please specify: [Textbox, 300] 

□ 
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B3. In [EAPSI_YEAR], what types of support did you (or your department or institution) provide 


to graduate students for the EAPSI application process? Check one response in each row 


Assistance with: 

YES 

NO 

DO NOT 
RECALL 

Locating an appropriate host institution or 
host researcher in an EAPSI site 

□ 

□ 

□ 

Contacting the host institution 

□ 

□ 

□ 

Application preparation 

□ 

□ 

□ 

Letter(s) of recommendation to host 
researcher in EAPSI site 

□ 

□ 

□ 

Letter(s) of recommendation to NSF 

□ 

□ 

□ 

Other. Please specify: [Textbox, 300] 

□ 




B4. Before your graduate student, [Fellow Name], applied to participate in the EAPSI program, 

were any of the following true? Check all that apply 

□ [1] I had been an EAPSI fellow GO TO B4a if this response checked regardless of other 
responses that may/may not be checked 

□ [2] I had been a graduate student, postdoc, faculty member, or visiting scientist in the 
foreign institution which hosted the Fellow GO TO B5 unless response option 1 also 
checked 

□ [3] I had a collaboration with the faculty member (or researcher) who hosted my 
graduate student advisee-^ GO TO B5 unless response option 1 also checked 

□ [4] I had a collaboration with another faculty member (or researcher) at the institution 
which hosted my graduate student advisee-^ GO TO B5 unless response option 1 also 
checked 

□ [5] I had been a graduate student, a postdoc, a faculty member, or a visiting scientist in 
[EAPSI_a ppSite] GO TO B5 unless response option 1 also checked 

□ [6] I had another connection to [EAPSI_appsite], Please specify: [Textbox, 500] GO 
TO B5 unless response option 1 also checked 

□ [7] I had been a graduate student, postdoc, faculty member, or visiting scientist in a 
country other than the United States GO TO B5 unless response option 1 also 
checked 

□ None of the above 


B4a. Were you an EAPSI fellow in [EAPSI_appSite]? Mark one answer 

□ Yes 

□ No, I was an EAPSI fellow in [EAPSI dropdown menu] set AdvisorSite = 
selection. 
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B5. Did you encourage [FELLOW NAME] to apply to the EAPSI program? Mark one answer 

□ Yes [GO TO B6] 

□ No [GO TO Cl] 

B6. What were the reasons you encouraged this Fellow to apply to the EAPSI program? Check 
all that apply 

□ My own positive experiences collaborating internationally 

□ The importance of gaining an international perspective 

□ To help the Fellow's future academic or professional career 

□ To foster more US - international collaborations among researchers 

□ Other. Please specify: [Textbox, 1000] 
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Module C: Your former student's participation in the EAPSI program 


Cl. While the Fellow was overseas at the host institution, did you provide any of the following 
ongoing support? Check all that apply 

□ I provided academic support (research planning, interpretation) to the Fellow 

□ I provided logistical, cultural or language support to the Fellow 

□ I helped facilitate collaboration between the Fellow and the host scientist (or the 
host scientist's research group) 

□ I worked with the applicant to develop a publication based on research conducted 
at the host institution 

□ I visited the Fellow at the host institution 

□ Other support: [Textbox, 1000] 

□ None of the above 


C2. In your view, how successful was the program in helping this Fellow to achieve the 
■Allowing? Check one response for each row 



Highly 

unsuccessful 

Somewhat 

Unsuccessful 

Somewhat 

Successful 

Highly 

successful 

1 cannot 
recall or 1 
do not 
know 

To establish a collaboration with 
a researcher(s) outside the U.S. 

□ 

□ 

□ 

□ 

□ 

To foster a long-lasting 
international collaboration 

□ 

□ 

□ 

□ 

□ 

To advance his/her research 
agenda 

□ 

□ 

□ 

□ 

□ 

To gain exposure to another 
country's research enterprise 

□ 

□ 

□ 

□ 

□ 

To learn a technique or an 
approach from the host scientist 

□ 

□ 

□ 

□ 

□ 

To gain access to resources or 
materials not widely available in 
the U.S. 

□ 

□ 

□ 

□ 

□ 

To become familiar with the 
culture and traditions of another 
country 

□ 

□ 

□ 

□ 

□ 


C2a. Did the program help the Fellow develop other skills or insights not listed above? [textbox, 
2500] 
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Module D: Benefits and challenges of Participation 


Dl. Upon the Fellow's return to the U.S., which of the following were true? Check all that 
apply 

□ I helped the Fellow integrate the research conducted at the host institution into 
their research conducted in the U.S. 

□ I advised the Fellow as he/she continued work begun during the EAPSI summer 

□ I assisted the Fellow in preparing a presentation and/or publication on research 
conducted at the host institution 

□ I provided the Fellow (or helped the Fellow obtain) financial support to continue 
work on the research conducted during the EAPSI summer 

□ I helped the Fellow maintain a professional collaboration developed at the host 
institution 

□ I helped the Fellow use the EAPSI experience to benefit his/her subsequent job 
search 

□ I helped the Fellow to pursue additional international research opportunities 

□ Other: Please describe: [Textbox, 500] 

□ None of the above 

D2. Which of the following benefits did you (and/or your research group) derive as a result of 
the EAPSI fellowship? Check all that apply. 

□ The Fellow introduced new knowledge, approaches, and/or technical skills 

□ The Fellow brought back samples or other materials unavailable or not easily accessible 
in the U.S. 

□ The Fellow's experience broadened our understanding of research enterprise in the 
foreign country 

□ The Fellow helped establish or maintain an important collaboration that has benefitted 
my research/my group's research 

□ The Fellow made my group and our research better known abroad 

□ The Fellow's experience changed the direction of our research or added a new 
dimension to our research 

□ We published papers which resulted from the Fellow's participation in EAPSI 

□ Other: Please specify. [Textbox, 1000] 

□ None of the above 


EAPSI ADVISOR SURVEY 


Page 13 


D3. What was one of the most important benefits (if any) to you or to your research group that 
resulted from having a graduate student participate in the EAPSI program? 

[Textbox, 2500] 

D4. Did any of the following occur as a result of the Fellow's participation in the EAPSI 
program? Check all that apply 

□ The Fellow became distracted from other important research projects 

□ The Fellow's degree completion was delayed 

□ The Fellow left our graduate degree program altogether 

□ Conflicts arose over intellectual credit for research conducted in the host site 

□ There was tension between the Fellow and the others members of my group due to 
his/her participation in the EAPSI program 

□ Other. Please specify. [Textbox, 1000] 

□ None of the above. 
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Module E: Benefit of EAPSI to the US department 


El. Which of the following characterized the Fellow's department at [Gradjnstitution] at the 
time that [Fellow Name] was enrolled ([EAPSI_YEAR])? Check all that apply. 

□ This department encouraged international collaborations 

□ Faculty in this department were rewarded for developing international research 
partnerships 

□ This department provided financial support to faculty pursuing international 
collaborations 

□ This department promoted (i.e., advertised) fellowships for graduate students or 
postdocs to conduct research in a foreign country 

□ This department hosted foreign post-docs and faculty visiting [Gradjnstitution] for 
research-related purposes 

□ I do not recall 

□ None of the above 

E2. As a result of this fellow's participation in the EAPSI program, did any of the following 
occur in this department (at [Gradjnstitution]) during the 2 year-period following this 
Fellow's [EAPSI_YEAR] summer fellowship? Check all that apply 

□ Graduate students in this department became more interested in collaboration with 
international researchers 

□ Faculty in this department became more interested in collaboration with 
international researchers 

□ Graduate students in this department began collaborating with international 
researchers 

□ Faculty in this department began collaborating with international researchers 

□ Collaborations with international researchers raised the prestige of the department 

□ Collaborations with international researchers helped attract students, postdoctoral 
fellows, and other researchers to the department 

□ Collaborations with international researchers helped bring additional funding to the 
department 

□ Administrative policies, procedures, or structures in the department or in the 
institution were established to facilitate collaboration with international 
researchers 

□ I do not recall ->GO TO E4 

□ Don't know, I was no longer at the department GO TO E4 

□ None of the above-^ GO TO E4 

E3. Feel free to elaborate on any of the above and/or to provide additional examples of how 
the fellow's participation in the EAPSI program affected this department at [Grad 
Institution], [Textbox. 2500] 
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E4. Do you think there have been any disadvantages for the department from having students 
participate in the EAPSI program? 

□ Yes Please specify. [Textbox, 2500] 

□ No 

□ I do not recall 

□ Don't know, I was no longer at the department 

E5a. Was the duration of the EAPSI program appropriate for a graduate student? 

□ Yes 

□ No. Please elaborate [textbox] 


E5b. Was the timing of the program appropriate (during the summer)? 

□ Yes 

□ No. Please elaborate [textbox] 


E6. Would you recommend (or have you recommended) the EAPSI program to other graduate 
students? Mark one answer 

□ I have recommended/would recommend the program. -> GO TO E6a 

□ I would not recommend the program. ->GOTOE6b 

□ I am not sure 


E6a. Why have you recommended (or why would you recommend) the EAPSI program to 
other graduate students? [Textbox, 2500] GO TO E7 


E6b. Why would you not recommend the EAPSI program to other graduate students? 
[Textbox, 2500] 


E7. Has your experience as faculty advisor to an EAPSI fellow made you more or less likely to 
get involved in international collaboration? Mark one answer 

□ I am more likely to get involved in an international research collaboration 

□ I am less likely to get involved in an international research collaboration 

□ The EAPSI experience has not affected my likelihood to get involved in an international 
research collaboration 
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Module F: Your Current Research Collaborations 


FI. Do you currently collaborate with researchers in countries outside the United States? 

□ Yes GOTO Fla 

□ No GOTO F2 


Fla. Who are these colleagues? Check all that apply 

□ The researcher who hosted my former graduate student ([Fellow Name]) in 
[EAPSI_actualSite] 

□ Other researcher(s) in [EAPSI_actualSite], 

□ One or more researchers in other foreign countries. 


Fib. What is the nature of your international colla 


aorations? Check all that apply 



With the 
Fellow's 
EAPSI host 

With another 
researcher in 
[EAPSI_actualSite] 

With researcher(s) 
in other foreign 
countries 

Co-author papers 

□ 

□ 

□ 

Co-author patent/license 
applications 

□ 

□ 

□ 

Co-sponsor professional 
conferences 

□ 

□ 

□ 

Share data or information 

□ 

□ 

□ 

Collaborate on a research 
project 

□ 

□ 

□ 

We send our graduate students 
and/or postdoctoral fellows to 
work in each other's 
laboratories/research sites 

□ 

□ 

□ 

Other. Please specify: [Textbox] 





F2. Which of the following are true? Check all that apply 

□ I have visited the host at the host's foreign research facility/institution 

□ The EAPSI host has visited me in the US 

□ A member of the EAPSI host's group visited me in the US or joined my group 

□ A foreign researcher connected to the EAPSI host, but not in the EAPSI host's group 
visited me in the US or joined my group 

□ A foreign researcher connected to the EAPSI host visited or joined another group in my 
department 

□ Because of the EAPSI program, I expanded my network of international collaborations in 
some other way. Please describe [textbox], 

□ None of the above 

F3. Please feel free to share any thoughts or recommendations about the EAPSI program. 

[Textbox, 2500] 


SUBMIT 

Submit button takes respondent to Thank You screen 
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EXIT SCREEN: 

If we have identified the wrong respondent or have erroneous information, display this screen 
and FLAG this respondent for Abt follow-up (weekly basis): 


EAPSI Graduate Advisor survey 


Please accept our apology. 

The information you supplied suggests that you are not eligible to participate in this study or 
that we have reached you in error. We regret any inconvenience to you. If you have any 
questions about this study or you would like to make a comment, please contact one of the 
following individuals: 

■ Dr. Alina Martinez, Study Director at Abt Associates: EAPSI_survey@abtassoc.com, or 
(866) 421-6223 (toll free within the U.S.); 

■ John Tsapogas, Office of International Science and Engineering, NSF: jtsapoga@nsf.gov. 

May we have permission to contact you by telephone to clarify your responses here? Entering 
your number does not obligate you to answer any questions. 

□ Yes, you may reach me at . [telephone digit entry] Go to Best Times. 

□ No, please do not contact me. 

Best time(s) to call ( check all that apply): 

□ Weekdays (9 to 5pm) 

□ Evenings (5 to 8pm) 

□ Saturday or Sunday 9 to 5pm 


THANK YOU SCREEN (FROM SUBMIT BUTTON) 


EAPSI host survey 


Conclusion 

Thank you for completing this survey. If you have any general comments about the survey, 
please write them below. 


Textbox, 2500 


If you have any questions about this survey or the study, please contact Dr. Alina Martinez, 
Study Director, Abt Associates, at (866) 421-6223 (may incur telephone charges if initiated 
outside the U.S.) or email her at EAPSI_survey@abtassoc.com. You may also contact John 
Tsapogas, who is overseeing this study at NSF, with any questions or comments: 
itsapoga@nsf.gov . 

Thank you for your assistance. We greatly appreciate your time and consideration. 
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Survey of Host Scientists for U.S. Graduate Students 
[LOCAL_PROGRAM_TEXT] 


LOCAL_PROGRAM_TEXT was automatically filled in the online version with one of the following: 

Summer Program in Japan 
Summer Institute in Taiwan 
Korea Summer Institute 
Summer Institute in China 

East Asia and Pacific Summer Institute (EAPSI) program in Singapore 
East Asia and Pacific Summer Institute (EAPSI) program in New Zealand 
East Asia and Pacific Summer Institute (EAPSI) program in Australia 

Programming notes (not displayed in the survey respondents saw online) appear in red or brown text 
throughout. 

Hyperlinked text appeared tfflbosstd wwtertftttd in this document but was not be formatted this 
way onscreen unless noted (underlining was used for email addresses, but embossing was not). 

General guidelines: 

Each screen displayed one question at a time unless otherwise indicated. 

Each screen displayed a standard set of Navigation buttons as shown below. 



mer Program in Japan Host Survey 


Module A: Verifying Information About You 

We would like to ask you about your experiences with hosting a graduate student supported by the Summer 
Program in Japan. Please respond to the best of your recollection. To begin, we 'd like to confirm that we 've 
reached the appropriate person. 

Please note: We will not reveal responses that you provide on any question in this survey to any 
of the students that you hosted in your institution. Neither will any data that can identify any 
specific student or host be shared with the sponsor of this survey (NSF) or the partner agency 
(or agencies) in Japan. 



Based on NSF's data about the Summer Program in Japan, your name is Kazuhiko Nishimura. Is this correct? 

Mark one answer 
© Yes, this is correct. 

© No, my name has changed or is misspelled above. My name is: 


Some items contained placeholders for fields that were programmed to be auto-filled from the sample 
file for each respondent. These fields are enclosed in [brackets] and printed in brown. 

Textboxes for open-ended responses are indicated with [textbox, 150] where the number indicates the 
length of the field (how many characters the respondent may type). 
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Sample variables: 


HOST_NAME: Name of the respondent, the host researcher. 

EAPSI_Site: Name of the country or site that co-sponsors the summer fellowship program. Values: 
Australia 
China 
Korea 
Japan 

New Zealand 

Singapore 

Taiwan 

EAPSI year the year that the respondent hosted the EAPSI fellow named in the survey. Values: 

2000, 2001, 2002, 2003 2004, 2005, 2006, 2007, 2008, or 2009 

Host institution the name of the university, research center, company where the host worked when 
EAPSI fellow visited. 

Fellow Name: name of individual EAPSI fellow who visited the respondent (host) 

Internal Variables: 

LOCAL_PROGRAM_TEXT: Sample file reads in EAPSI_Site and LOCAL_PROGRAM_TEXT takes a value 
as follows 

If EAPSI_SITE = "Japan" then LOCAL PROGRAM TEXT = "Summer Program in Japan" 

Else if EAPSI_SITE = "Taiwan" then LOCAL PROGRAM TEXT = "Summer Institute in Taiwan" 

Else if EAPSI_SITE = "Korea" then LOCAL PROGRAM TEXT = "Korea Summer Institute" 

Else if EAPSI SITE = "China" then LOCAL PROGRAM TEXT = "Summer Institute in China" 

Else if EAPSI_SITE = "Singapore" then LOCAL PROGRAM TEXT = "East Asia and Pacific Summer 
Institute (EAPSI) program in Singapore" 

Else if EAPSI_SITE = "New Zealand" then LOCAL PROGRAM TEXT = "East Asia and Pacific 
Summer Institute (EAPSI) program in New Zealand" 

Else if EAPSI_SITE = "Australia" then LOCAL PROGRAM TEXT = "East Asia and Pacific Summer 
Institute (EAPSI) program in Australia" 

PROGRAM: Sample file reads in EAPSI_Site and PROGRAM takes a value as follows 

If EAPSI SITE = "Japan" then PROGRAM = "Summer Program in Japan" 

Else if EAPSI SITE = "Taiwan" then PROGRAM = "Summer Institute in Taiwan" 

Else if EAPSI SITE = "Korea" then PROGRAM = "Korea Summer Institute" 

Else if EAPSI_SITE = "China" then PROGRAM = "Summer Institute in China" 

Else if EAPSI_SITE = "Singapore" then PROGRAM = "EAPSI program in Singapore" 

Else if EAPSI SITE = "New Zealand" then PROGRAM = "EAPSI program in New Zealand" 

Else if EAPSI_SITE = "Australia" then PROGRAM = "EAPSI program in Australia" 
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Survey of Host Scientists for U.S. Graduate Students 
[LOCAL_PROGRAM_TEXT] 


PROGRAMMER NOTE: LOCAL_PROGRAM_TEXT, PROGRAM, and EAPSI_site values should appear where 
indicated in place of [brackets] below: 

Welcome and thank for your interest in this study. This survey is being conducted by Abt Associates Inc. and 
our subsidiary, AbtSRBI, for the National Science Foundation (NSF), to learn about the experiences of individuals 
who participated in the [LOCAL_PROGRAM_TEXT]. This survey will give NSF information about the international 
collaborations of U.S. graduate students and help NSF improve programs intended to foster a globally engaged 
scientific and engineering workforce. You are receiving this survey because you are listed in NSF's program 
database as a host of one or more U.S. graduate students ([PROGRAM] "fellows"). We estimate that it will take 
approximately 15 minutes to complete the survey. 

Confidentiality and Participation 

Participation in the survey is voluntary and nonparticipation will have no impact on you or your institution. You 
may skip questions on the survey or discontinue participation at any time. There are minimal risks associated 
with your participation. We take your privacy very seriously. Your responses to this survey will be protected 
under the U.S. Privacy Act. There is minimal risk of breach of confidentiality, and we have put in place 
procedures to minimize this risk. Reports will never identify you by name, and information from the study will 
only be reported in the aggregate at the program level, combined with about 500 other responses. We will not 
reveal responses that you provide on any question in this survey to any of the fellows that you hosted in your 
institution. Neither will any data that can identify any specific fellow or host be shared with the sponsor of this 
survey (NSF) or the partner agency (or agencies) in [EAPSI site] . When we receive your survey we will detach 
and store separately your name and other identifying information that could be used to link you to your survey 
responses. Survey responses will be stored on a secure drive that is only accessible to members on the study 
team. Only study team web technicians and data analysts from Abt Associates and AbtSRBI will see individual 
responses that can be linked to you. Survey data files will be shared with NSF at the end of the study, only after 
study team members have examined the data to be free of any information that could help identify you; this 
cleaning includes procedures to prevent someone from inferring your identity by analyzing non-identifying data. 
Hence, we encourage you to respond candidly about your experiences. Separate from your individual responses 
to the survey we will provide NSF any updated contact information we have found or requested from you. None 
of this contact information will be linked in any way to your survey responses. At the conclusion of the study, 
Abt Associates and AbtSRBI will destroy all records, electronic or otherwise, that link you to your survey 
responses. 

Questions 

If you have questions about the study, please contact the study director, Alina Martinez of Abt Associates Inc. at 
(866) 421-6223 (may incur international telephone charges if initiated outside the U.S) or email her at 
EAPSI_survey@abtassoc.com. You may also contact the evaluation's program officer at NSF, John Tsapogas 
(jtsapoga@nsf.gov). If you have questions about your rights as a research participant, you may contact Teresa 
Doksum, the Abt Institutional Review Board Administrator at (877) 520-6835 (may incur international telephone 
charges if initiated outside the U.S) or by email: irb@abtassoc.com. To learn more about this study, please refer 
to thefcreqwefttty Asked Q)Uestte«S page. 

Consent 

Please click on "Begin" if you agree to participate in this study. pGIN] 

This study's IRB approval number is #0494, valid from 8/6/2010 to 8/5/2011. For questions, please contact 
Teresa Doksum, IRB Administrator, Abt Associates, at IRB@abtassoc.com. The valid OMB control no. for this 
information collection is 3145-0214. (Expires on 12/31/13). 
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Navigating through the survey: 

As you work through the survey, your responses are automatically saved. You may change a response 
by clicking on the PREVIOUS ITEM button. Use theNEXT ITEM button to advance to the next 
question. At any time, you close your browser if you wish to temporarily pause the survey and return 
to it at a later time. On each page of the survey, you may click on the 'fAQs* button if you have a 
question during the survey or need information about how to contact the survey administrator. 

When you have completed the survey, please click on theSUBMIT button at the end of the survey. 
You may submit the survey even if there are some questions that you choose not to answer. 

CONTINUE 
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FAQs are Optional screens, displayed only if R clicks on Frequently Asked Questions button. 
Programmer note: Note that variables below must be auto-completed in Items 1, 2, 3, 4, 6 and 10 

FREQUENTLY ASKED QUESTIONS 


• What is the [PROGRAM]? 

• Why are you doing this study? 

• Why have I been selected to participate in this survey? 

• How did you get my contact information? 

• How long will this survey take to complete? 

• How will you use my comments? 

• What happens if I don’t answer this survey? 

• Does this study have human subjects review clearance? 

• Who is funding the study? 

• Who are Abt Associates Inc and AbtSRBI? 


What is the [PROGRAM]? 

The [LOCAL PROGRAM TEXT] is a program in which the National Science Foundation 
(NSF) and its partner organizations in Australia, China, Japan, Korea, New Zealand, 

Singapore, and Taiwan cooperate to provide U.S. graduate students an opportunity to spend 
eight to ten weeks collaborating with a host researcher at a university or research institute in 
one of these locations. The program matches accomplished researchers in the host location 
with promising U.S. graduate students to foster international collaboration. 

Why are you doing this study? 

NSF is interested in learning about the experiences of researchers who have hosted one or more 
graduate students from the United States as part of the [PROGRAM]. In particular, NSF 
would like to understand how host researchers perceive the program and the students they have 
hosted, and what kinds of benefits and challenges host researchers experienced. The 
information collected in the study will help NSF make improvements to the program, and 
understand how best to support and encourage international collaboration. 

Why have I been selected to participate in this survey? 

You have been selected to participate because we have identified you as having hosted one or 
more U.S. graduate students participating in the [PROGRAM]. 

How did you get my contact information? 

We identified you from records maintained by the National Science Foundation on 
[PROGRAM] participants and the scientists who hosted them. We then confirmed your contact 
information through an internet search. 

How long will this survey take to complete? 

We estimate that the survey will take about 15 minutes. 
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How will you use my comments? 

Responses from all survey respondents will be used to answer questions about the experiences of 
EAPSI host researchers with the program and with the guest students. We will not reveal 
responses that you provide on any question in this survey to any of the students that you hosted 
in your institution. Neither will any data that can identify any specific student or host be shared 
with the sponsor of this survey (NSF) or the [PROGRAM] partner agency in [EAPSI_site] . 

What happens if I don’t answer this survey? 

Participation in the survey is voluntary. Abt Associates will not share information with the 
National Science Foundation or its partner agencies in your location about your decision to 
participate or not to participate. 

Does this study have human subjects review clearance? 

Yes, the study was approved by Abt Associates’ Institutional Review Board. If you have any 
concerns about your participation in this survey, please contact Teresa Doksum, Institutional 
Review Board Administrator at Abt Associates, at (877) 520-6835 or via email at 
irb@abtassoc.com. A call to this number may incur international charges. 

Who is funding the study? 

The study has been funded by the National Science Foundation under contract 
GS-10F-0086K. Abt Associates and AbtSRBI will complete the study. 

Who are Abt Associates Inc and AbtSRBI? 

Abt Associates is an independent research firm headquartered in Cambridge, Massachusetts. 
AbtSRBI is a wholly-owned subsidiary of Abt Associates specializing in large-scale data 
collection and public opinion research. NSF has contracted with Abt Associates and AbtSRBI to 
design and implement a survey of researchers who hosted a U.S. graduate student participating in 

[PROGRAM], 
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EAPSI Host Survey 


Module A: Verifying Information About You 

We would like to ask you about your experiences with hosting a graduate student supported by the 
[PROGRAM], Please respond to the best of your recollection. To begin, we'd like to confirm that we've 
reached the appropriate person. 

Please note: We will not reveal responses that you provide on any question in this survey to any of the 
students that you hosted in your institution. Neither will any data that can identify any specific 
student or host be shared with the sponsor of this survey (NSF) or the partner agency (or agencies) in 
[EAPSI_site]. 

Al. Based on NSF's data about the [PROGRAM], your name is [HOST_NAME], Is this correct? Mark one 
answer 

□ Yes, this is correct. 

□ No, my name has changed or is misspelled above. My name is: [Textbox, 75] Set HOST_NAME = 
entered text. 

□ No, I am not the person named above. I believe you have reached me by mistake. EXIT SURVEY 

A3. Based on NSF data, you hosted one or more [PROGRAM] fellows from the United States most 
recently in [EAPSI_Year], Is this information correct? Mark one answer 

□ Yes, the year is correct. 

□ No, the year is incorrect. The correct year is (enter 4-digit year): [Textbox, yyyy] Set EAPSI_Year 
= entered year 

□ I do not recall the exact year 

A4. Based on NSF's data about the [PROGRAM], you were at [Host institution] in [EAPSI_YEAR], Is this 
information correct? Mark one answer 

□ Yes, the institution is correct. 

□ No, the institution is incorrect. The correct institution is: [Textbox, 150] Set Host institution = 
entered text 

A5. During the week of October 1, 2010, were you working at [Host institution], the institution where 
you worked in [EAPSI_YEAR]? Mark one answer 

□ Yes, I was working at this institution then 

□ No, I no longer work there. I am now working at (please tell us the name of your current 
employer, university, or company): [Textbox, 150] 

A5a. As of October 1, 2010, approximately what percentage of your current research group was from 
other countries (that is, places other than [EAPSI_SITE])? Mark one answer 

□ 0 - 10 % 

□ 11-25% 

□ 26-50% 

□ 51-75% 

□ >75% 

□ Not applicable, I do not work in a group 
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A6. Which of the following characterizes the department (or research unit within an institute) at [HOST 

INSTITUTION], where you most recently participated as a [PROGRAM] host? Check all that apply. 

□ The department (or research unit) where I was a [PROGRAM] host encouraged faculty (or 
employees) to work with graduate students from the United States 

□ The department (or research unit) where I was a [PROGRAM] host encouraged international 
collaborations 

□ Faculty (or employees) in the department (or research unit) where I was a [PROGRAM] host 
were rewarded for developing international research partnerships 

□ The department (or research unit) where I was a [PROGRAM] host provided financial support to 
faculty (or employees) pursuing international collaborations 

□ The department (or research unit) where I was a [PROGRAM] host promoted (that is, 
advertised) fellowships and other opportunities for researchers in my country to conduct 
research in another country 

□ The department (or research unit) where I was a [PROGRAM] host hosted foreign researchers 
visiting my institution for research-related purposes 

□ None of the above 

□ I do not recall 

A2. Based on NSF's data about the [PROGRAM], at one time you hosted a graduate student from the 

United States named [Fellow name]. Is this information correct? Mark one answer 

□ Yes, this is correct. 

□ I do not recall this student by name, but I have hosted U.S. graduate students as part of the 
[PROGRAM], GO TO A2a. 

□ No, I did not host this person-or any other graduate student from the United States. EXIT 
SURVEY. 

□ I never participated in [PROGRAM] for U.S. graduate students. EXIT SURVEY. 

A2a. Although you do not recognize the former [PROGRAM] fellow, "[FELLOW NAME]," NSF's data 
about [PROGRAM] indicate that you hosted a graduate student from the United States (or more 
than one student) as part of the [PROGRAM] in [EAPSI_YEAR], We would appreciate it if you 
would continue with the survey. Please answer the questions that follow based on your most 
recent experience hosting U.S. graduate students in the [PROGRAM], Please click on "Next Item" 
to continue. GO TO ITEM E4 
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Module B: Becoming a Host to a U.S. Graduate Student 


The remaining items on this survey refer to the [PROGRAM] Fellow named above ([Fellow name]) or to 
the weeks ([EAPSI_YEAR]) during which you hosted this individual at your institution 
([HOST_INSTITUTION]). If you have hosted more than one [PROGRAM] Fellow, please answer items 
on this survey with respect to [Fellow name], who you hosted in [EAPSI_ Year], 

B2. Why did you decide to host this [PROGRAM] Fellow ([Fellow Name])? Check all that apply 

□ I was interested in the project proposed by the Fellow 

□ I was interested in establishing or maintaining collaboration with a US researcher 

□ I personally knew, knew of, or previously collaborated with the Fellow 

□ I personally knew, knew of, or previously collaborated with the Fellow's graduate advisor 

□ I personally knew, knew of, or previously collaborated with researchers at the Fellow's 
institution 

□ I had a positive experience with the [PROGRAM] in the past 

□ I had a positive experience with another international program 

□ To improve the status of my department and/or institution 

□ To create an international environment in my research group 

□ To attract students/postdocs to my research 

□ To learn new methodologies, approaches, or tools from the Fellow 

□ In my field, individuals trained at U.S. graduate institutions are highly sought-after 

□ My research area is particularly suitable for international collaboration 

□ Other: [Textbox, 300] 

B3. Did you have any of the following concerns about hosting this Fellow ([FELLOW_NAME])? Check all 
that apply 

□ The Fellow's proposed project was especially risky 

□ The length of the [PROGRAM] was too short for the proposed project 

□ I was concerned about the fellow's level of commitment to a collaboration with me 

□ I was concerned about the risks of international collaboration in general 

□ I had a negative experience with the [PROGRAM] in the past 

□ I had a negative experience with other U.S. graduate students fellows (not affiliated with the 
[PROGRAM]) 

□ I was concerned about the integrating this student into my research group 

□ I was concerned that I might not (or my research group might not) benefit from hosting this 
student 

□ In my field, individuals trained at U.S. graduate institutions sometimes have gaps in their 
knowledge, skills, or abilities 

□ My research area is not particularly suitable for international collaboration 

□ Other, please specify: [Textbox, 300] 
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B4. Before hosting [Fellow Name] in [EAPSI_YEAR], had you ever before hosted a visiting graduate 
student from the United States? Mark one answer. 


□ Yes, I had hosted another [PROGRAM] Fellow or Fellows from the United States previously 

□ Yes, but the graduate student(s) was not sponsored by the [PROGRAM] 

□ No, I had never before hosted visiting graduate students from the United States 


B5. Prior to your participation as a [PROGRAM] host, had you visited the United States for educational, 
research, or other professional purposes? Check all that apply 

□ Yes, I was an undergraduate student in the United States 

□ Yes, I was a graduate student in the United States 

□ Yes, I was a postdoctoral fellow in the United States 

□ Yes, I was a visiting scientist in the United States 

□ Yes, I was a faculty member in the United States 

□ Yes, I came to the United States for a conference, a workshop, or a meeting 

□ Other reason for visiting the United States. Please specify: [Textbox, 300] 

□ No, before participating as a [PROGRAM] host I had not visited the United States for any 
professional purposes 

B6. Did you know [Fellow name] or one of his/her colleagues before you hosted this [PROGRAM] fellow? 
Mark one answer 

□ Yes, I knew this fellow before he (or she) came to [EAPSI_site] 

□ I did not know the fellow but I knew one (or more) of the fellow's graduate faculty advisors 

□ I knew the fellow and one or more of the fellow's graduate faculty advisors 

□ No, I did not know the fellow or any of the fellow's graduate advisors 
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B7. How would you characterize the ease or difficulty of the following aspects of arranging the Fellow's 
visit? 



Easy 

Somewhat 

difficult 

Very 

Difficult 

Not applicable 
(1 did not 
provide this or 
make these 
arrangements 
for the Fellow) 

Providing supporting materials for the 
Fellow's application to NSF 

□ 

□ 

□ 

□ 

Providing assistance with obtaining a visa 

□ 

□ 

□ 

□ 

Arranging the Fellow's lodging 

□ 

□ 

□ 

□ 

Arranging the Fellow''s office space 

□ 

□ 

□ 

□ 

Arranging the Fellow's access to buildings, 
laboratories, or to other institutional 
facilities 

□ 

□ 

□ 

□ 

Finding an appropriate research project 

□ 

□ 

□ 

□ 


B7a. Please describe any details you would like to share about any challenging aspects of arranging the 
Fellow's visit. [Textbox, 2500] 
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Module C: Your Experiences as a [PROGRAM] Host 


Cl. During this [PROGRAM] Fellow's ([FELLOW_NAME]) visit to your institution, which of the following 
types of people worked directly with the visiting fellow on research? Check all that apply 

□ I worked directly with the fellow 

□ Research scientists/engineers in my research group worked directly with the fellow 

□ Postdoctoral fellows in my research group worked directly with the fellow 

□ Graduate students in my research group worked directly with the fellow 

□ Undergraduate students in my research group worked directly with the fellow 

□ Other (please describe): [textbox, 300] 

□ The visiting fellow worked directly with other visiting [PROGRAM] fellows 

□ The visiting student worked independently without supervision 

□ Other (please specify): [Textbox, 300] 

□ None of the above 


C2. In [EAPSI_YEAR], what was the size of your research group at [Host Institution]? Do not include 
visiting Fellows in your answer. Mark one answer 

□ 1 person (myself) 

□ 2-5 people 

□ 6-10 people 

□ 11-20 people 

□ >20 people 

□ I do not remember 

C2a. In [EAPSI_YEAR], approximately what percentage of your research group was from other 

countries (besides [EAPSI_SITE])? Do not include visiting Fellows in your answer. Mark one 
answer 
□ 0 - 10 % 

□ 11-25% 

□ 26-50% 

□ 51-75% 

□ >75% 

□ I do not recall 

□ Not applicable, I did not work in a research group 
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C3. In [EAPSI_YEAR], who was primarily involved in the following activities on which you and the 
[PROGRAM] Fellow ([FELLOW_NAME]) worked? 



Mostly the 
Fellow 

independently 
(or the Fellow 
and his/her 
U.S. advisor) 

Mostly me 
or members of 
my group 
without the 
Fellow 

The Fellow and 
1 (and/or 
members of 
my research 
group) 

together about 
equally 

Not applicable 
(this activity 
was not part of 
the research) 

Developing the ideas, hypotheses, 
broad framework, or vision for the 
research project 

□ 

□ 

□ 

□ 

Researching literature or research base 
relevant to the project 

□ 

□ 

□ 

□ 

Keeping records, tracking supplies, 
resources, 

□ 

□ 

□ 

□ 

Developing instrumentation, software, 
equipment, or data collection processes 

□ 

□ 

□ 

□ 

Collecting data or carrying out 
simulations 

□ 

□ 

□ 

□ 

Analyzing data or observations 

□ 

□ 

□ 

□ 

Interpreting results 

□ 

□ 

□ 

□ 

Planning or developing follow-up work 
based on results 

□ 

□ 

□ 

□ 

Written, oral dissemination of results 
(publications, presentations) 

□ 

□ 

□ 

□ 
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C4. Do you agree or disagree with the following aspects of your participation as host to a [PROGRAM] 


Fellow? 



Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

Does not 
apply 

The Fellow integrated with staff/members of my 
research group 

□ 

□ 

□ 

□ 

□ 

The Fellow had sufficient knowledge and expertise to 
be a full participant in a research collaboration with 
me 

□ 

□ 

□ 

□ 

□ 

The amount of time for the fellowship experience 
was too short 

□ 

□ 

□ 

□ 

□ 

Language was a barrier to the Fellow's ability to 
interact with me and/or my group 

□ 

□ 

□ 

□ 

□ 

The Fellow was willing to take appropriate risks 
necessary for research 

□ 

□ 

□ 

□ 

□ 

The Fellow exercised appropriate caution in his/her 
approach to research 

□ 

□ 

□ 

□ 

□ 

The Fellow was hard-working and dedicated 

□ 

□ 

□ 

□ 

□ 

The Fellow spent sufficient time working on the 
project 

□ 

□ 

□ 

□ 

□ 

Scientific cultures of our countries are similar, making 
productive collaboration possible 

□ 

□ 

□ 

□ 

□ 

Differences in lifestyles and general cultures of our 
countries are a barrier to collaboration. 

□ 

□ 

□ 

□ 

□ 


C6. Which of the following were true about you and the EAPSI Fellow [Fellow Name]? Check all that 
apply 

□ Our research interests were well-matched 

□ Our work styles were well-matched 

□ Our goals and expectations for the experience were similar 
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Module D: Outcomes of Participation as a [PROGRAM] Host 

Dl. Since the conclusion of this fellow's summer research with you, have you collaborated or 
communicated further with [Fellow Name]? Mark one answer 

□ I have collaborated further with this individual on research GO TO Dla 

□ I have communicated further with this individual, but we have not collaborated further 

GOTO D2 

□ I have not communicated further with this individual GOTO D2 

Dla. How recently has the latest collaboration occurred? Mark one answer. 

□ Within the past 6 months 

□ Within the past 12 months 

□ 1-2 years ago 

□ 3 or more years ago 

Dlb. What was/is the nature of your collaboration? Check all that apply. 

□ I have a position in this individual's current institution (within the U.S.) 

□ This individual has a position at my current institution (outside the U.S.) 

□ We have exchanged ideas, data, research results, or tools 

□ We have co-authored research papers together 

□ We have co-advised students together 

□ We have visited each other at our institutions 

□ Other: [Textbox, 300] 

GOTO D3a 

D2. Why do you no longer collaborate with this former fellow? Check all that apply 

□ Our research interests diverged 

□ One or both of us lacked funding needed to maintain collaboration 

□ Language differences have hindered further collaboration 

□ Political or cultural differences have hindered further collaboration 

□ Geographic distance has hindered further collaboration 

□ I did not think that further collaboration would be beneficial for me 

□ The [PROGRAM] fellow did not actively pursue or maintain further collaboration with me 

□ One (or both) of us is too busy with other projects 

□ Other - specify [textbox, 300] 

D3a. Are you currently collaborating with any of this [PROGRAM] fellow's colleagues? Check all that 
apply 

□ With this individual's former faculty advisor 

□ With graduate students who currently work with this former [PROGRAM] fellow 

□ With postdoctoral fellows who currently work with this former [PROGRAM] fellow 

□ With other researchers working with this former [PROGRAM] fellow 

□ With none of the above 

□ I'm not sure if my collaborating partner is a colleague of this former [PROGRAM] fellow 

D3b. Are you currently collaborating with other U.S. researchers? 

□ Yes-^GO TO D3d 

□ No^ GO TO D4 
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D3d. Have your collaborations with any U.S. researchers resulted from your participation as a 
[PROGRAM] host? Mark one answer 

□ Yes 

□ No 

D4. Please indicate the number of the following which you published or produced in collaboration with 
your former [PROGRAM] fellow, [Fellow Name] 


Total number 
published/ 
in-press with this 
former [PROGRAM] 
fellow 



Peer-reviewed journal articles 


Peer-reviewed conference publications (e.g. 
abstracts, conference papers, posters) 


Patents, registered or pending 


Book chapter(s) (e.g., in edited volumes) 


D4a. In what year did you most recently publish a paper or register a patent with this fellow? (enter 4- 
digit year): [Textbox, yyyy] 


EAPSI Foreign Host Survey 


Page 16 


Module E: Satisfaction with Hosting [PROGRAM] fellow/fellows 

We will not reveal responses that you provide on any question in this survey to any of the fellows that 
you hosted in your institution. Neither will any data that can identify any specific fellow or host be 
shared with the sponsor of this survey (NSF) or with the [PROGRAM] partner agency in [EAPSI_site], 

El. On a scale of 1-5, with 1 being "much less satisfied," and 5 being "much more satisfied," how 

satisfied were you overall with this [PROGRAM] fellow [Fellow Name] compared to other graduate 
students? Mark one answer. 


® 

© 

© 

© 

© 

Much less 
satisfied than 1 
have been with 
other graduate 
students 

Somewhat less 
satisfied than 1 
have been with 
other graduate 
students 

Equally satisfied 

as 1 have been 
with other 
graduate 
students 

Somewhat more 
satisfied than 1 
have been with 
other graduate 
students 

Much more 
satisfied than 1 
have been with 
other graduate 
students 


If you cannot select a response because you have not worked with any other graduate students, 
please check here: □ If Respondent checks this box no rating should be allowed 


Ela. What did you find most satisfying or unsatisfying about this fellow or about the [PROGRAM] 
experience as a whole? [Textbox, 2500] 
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E2. Which of the following benefits (if any) did you derive as a result of your participation in the 
[PROGRAM]? Check all that apply 

□ Hosting [PROGRAM] fellows was the first opportunity I had to work with US graduate students 

□ I established or renewed a collaboration with other US researcher/researchers 

□ I published (or researchers in my group published) research papers based on the collaborative 
work with this [PROGRAM] Fellow 

□ I obtained (or researchers in my group obtained) funding based on the collaborative work 

□ I gave (or researchers in my group gave) one or more presentations based on the collaborative 
work 

□ I learned (or researchers in my group learned) new methodological/analytical techniques or 
theoretical approaches 

□ I obtained (or researchers in my group obtained) access to resources not easily available at my 
institution or location 

□ I became (or researchers in my group became) more familiar with the research enterprise in the 
United States 

□ Participation in the program changed the direction of some research projects in my group 

□ Participating in the program helped advance my career 

□ Participating in the program enhanced my interest in collaborating with US researchers 

□ Participating in the program helped me recruit other graduate students or postdoctoral fellows 

□ Hosting a [PROGRAM] Fellow (or several Fellows) enhanced the recognition of my work by peers 

□ I improved my English language skills 

□ Other benefits (please describe): [textbox, 300] 

□ None of the above 

E3. Did you encounter any of the following challenges when hosting a [PROGRAM] fellow? Check all 
that apply 

□ The fellow's lack of familiarity with the primary language spoken here made collaboration more 
difficult than anticipated 

□ The fellow and I had differences of opinion about the direction of research 

□ The fellow had unanticipated gaps in his/her preparation to conduct research with me 

□ The fellow did not devote enough time/effort to the research collaboration 

□ The fellow lacked sufficient understanding of cultural norms in my country 

□ The fellow was disrespectful, caused conflict within my research group 

□ The fellow needed too much guidance 

□ The fellow worked too independently, did not work well as a collaborator or team member 

□ Other challenges; please specify: [textbox, 300] 

□ None of the above 
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E4. Have any of the following taken place at your institution as a result of your participation as host to a 
[PROGRAM] Fellow? Check all that apply 

□ My colleagues increased their own collaborations with US researchers 

□ Administration in my department (or research unit) became more supportive of collaboration 
with US researchers 

□ Additional policies, procedures, or structures have been put in place at my institution to 
facilitate international collaboration 

□ Collaborations with US researchers helped attract students and other researchers to my 
institution 

□ Other changes at my institution as a result of hosting [PROGRAM] Fellow or Fellows (please 
describe): [textbox, 1500] 

□ None of the above 

E4a. Can you provide any examples of how participating in the [PROGRAM] affected your institution? 

Please mark "No" if your participation as a [PROGRAM] host had no effect on your institution. 

[Textbox, 2500] □ No. GO TO E5a. 


E5a. What were the best aspects of your experience with the [PROGRAM]? [Textbox, 2500] 

E5b. What were the most challenging aspects of your experience with the [PROGRAM]? [Textbox, 2500] 

E6. Were there any unexpected outcomes of your participation in the [PROGRAM] experience? 
[Textbox, 2500] 


Programmer note: Item E7a (or Item E7b) should appear on same screen as Item E7 if user selects response 
option (1) or (2) 

E7. Would you recommend (or have you recommended) to others that they host [PROGRAM] fellows? 
Mark one answer 

□ I would recommend (or I have recommended) the program. GO TO E7a 

□ I have not recommended the program, and I would not recommend the program. GO TO E7b 

□ I am not sure GOTO E7c 

E7a. Why? [Textbox, 2500] GO TO E7c 
E7b. Why not? [Textbox, 2500] GO TO E7c 


E7c. Based on your experience hosting one or more graduate students from the United States, what 
recommendations would you make to other U.S. graduate students about working with 
researchers in your country? [Textbox, 2500] 


E8. Please feel free to share any additional thoughts or recommendations about [PROGRAM], [Textbox, 
2500] 


SUSMIT 

Submit button takes respondent to Thank You screen 
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EXIT SCREEN: 

If we have identified the wrong respondent or have erroneous information, display this screen and 
FLAG this respondent for Abt follow-up. 


host survey 


Please accept our apology. 

The information you supplied suggests that you are not eligible to participate in this study or that we 
have reached you in error. We regret any inconvenience to you. If you have any questions about this 
study or you would like to make a comment, please contact one of the following individuals: 

■ Dr. Alina Martinez, Study Director at Abt Associates: EAPSI_survey@abtassoc.com or (866) 421- 
6223 (may incur telephone charges if initiated outside the U.S.); 

■ John Tsapogas, Office of International Science and Engineering, NSF: jtsapoga@nsf.gov. 

May we have permission to contact you by email in order to clarify your responses here? Answering 
"yes" does not obligate you to answer any questions. 

□ Yes, you may contact me. My preferred email address is: [textbox 150] 

□ No, please do not contact me. 

Thank you very much. 


THANK YOU SCREEN (FROM SUBMIT BUTTON) 


host survey 


Conclusion 

Thank you for completing this survey. If you have any general comments about the survey, please write 
them below. 


Textbox, 2500 


If you have any questions about this survey or the study, please contact Dr. Alina Martinez, Study 
Director, Abt Associates, at (866) 421-6223 (may incur telephone charges if initiated outside the U.S.) or 
email her at EAPSI_survey@abtassoc.com. You may also contact John Tsapogas, who is overseeing this 
study at NSF, with any questions or comments: itsapoga@nsf.gov . 

Thank you for your assistance. We greatly appreciate your time and consideration. 
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Appendix F: Benchmark to National Data 


To place the outcomes of all EAPSI applicants with the broader context of national science and 
engineering degree holders, the outcomes of EAPSI fellows and all EAPSI applicants were compared 
to national estimates for science and engineering graduates. Specifically, the outcomes of EAPSI PhD 
graduates were compared against national estimates from the Survey of Doctoral Recipients (SDR), 
and the outcomes of EAPSI master's graduates were compared against estimates of individuals who 
had completed master's degrees in the National Survey or Recent College Graduates (NSRCG). 

The 2006 and 2008 SDR was used to compare the EAPSI applicants, whose highest degree is a 
doctorate, to those of a nationally representative sample of science, engineering and health (SEH) 
doctoral degree recipients on key employment, postdoctoral appointment, and international 
collaboration variables. The 2006 and 2008 NSRCG was used to compare the EAPSI applicants, 
whose highest degree is masters, to a nationally representative sample of SEH master's degree 
recipients on key employment, and international collaboration variables. 

These comparisons are descriptive in nature and are not intended to address the impacts of the 
EAPSI program. Thus, findings should be interpreted with an understanding that there may be 
uncontrolled initial differences between EAPSI applicants and those who responded to the SDR and 
the NSRCG. 

In these benchmarking analyses covariates are grand mean centered using the SDR and EAPSI 
samples (or the NSRCG and the EAPSI samples). Details of the sampling frame and how the SDR data 
(2006 and 2008 waves) and the analyses conducted are provided in Appendix D. 


Benchmark Comparison of EAPSI Applicants to National Samples 


EAPSI applicants were compared to the national samples on the frequency with which they worked 
with individuals in other countries (Exhibit F.l and Exhibit F.2). EAPSI PhD applicants were more 
likely to be working "for pay or profit" (almost 100 percent of EAPSI applicants with PhDs versus 94 
percent of the SDR sample, a statistically significant difference). Both EAPSI applicants with PhDs 
and those with master's degrees were significantly more likely to be working with individuals in 
other countries, compared to their counterparts nationally (77 percent of EAPSI PhD applicants 
versus 31 percent of the SDR sample, and 32 percent of EAPSI applicants with master's degrees 
compared to 21 percent of master's degree holders in the NSRCG). 
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Exhibit F.l: Work and International Collaborations of EAPSI Doctoral Degree Applicants and SDR 
Sample during Reference Week 



*p<0.05, **p<0.01, ***p<0.001 

NOTES: Working for pay/profit: This item was answered by former EAPSI applicants who had earned a PhD as of October 1, 
2010 (N=642 Missing=0) and by SDR 2008 respondents who had completed a PhD by October 1, 2008 (N=29,974 
Missing=0). Worked with individuals in other countries: This item was answered by former EAPSI applicants who had 
earned a PhD and were employed during the week of October 1, 2010 (N=615 Missing=19) and by SDR 2006 respondents 
who had completed a PhD and were employed the week of April 1, 2006 (N=27,119 Missing=0). This item was not 
included in the SDR 2008 wave. 

SOURCE: EAPSI Applicant Survey-Items Dla, Die, D2 and D4 and SDR Survey 2008-Items A1 and A27 


Exhibit F.2: International Collaborations of EAPSI Master's Degree Applicants and NSRCG Sample 
during Reference Week 


Worked with individuals in other 
country 



■ EAPSI 

■ NSRCG 


*p<0.05, **p<0.01, ***p<0.001 

NOTES: The NSRCG sample excluded individuals living outside of the United States. This item was answered by EAPSI 
applicants whose highest degree earned was a master's degree, and were employed during the week of October 1, 2010 
(N=222 Missing=0) and by NSRCG respondents who had completed a master's degree and were employed as of April 1, 
2006 (N=3,973 Missing=0). 

SOURCE: EAPSI Applicant Survey-Items Dla, Die, and D4. NSRCG Survey 2006-Items B1 and B27. This item was not 
included in the NSRCG 2008 survey wave. 


Additional comparisons were conducted to investigate the differences in the work settings between 
EAPSI applicants with PhDs and national estimates for similar S&E graduates (Exhibit F.3). EAPSI 
applicants were more likely to be employed in an educational institution (74 versus 46 percent) and 
less likely in state or local government (1 versus 3 percent). 
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Exhibit F.3: Employers of EAPSI Doctoral Degree Applicants and SDR Sample during Reference 
Week 


Educational institution 


Self-employed/business owner 


Private sector 


Local/state government 


Federal government 



■ EAPSI PhD 

■ SDR 


*p<0.05, **p<0.01, ***p<0.001 

NOTE: Reference week was October 1, 2010 for EAPSI applicants and October 1, 2008 for the SDR sample. 

This item was answered by ail former EAPSI applicants who had earned a PhD and were employed as of October 1, 2010 
(N= 378 Missing=0) and by SDR 2008 respondents who had completed a PhD and who were employed during the week of 
October 1, 2008 (N=26,191 Missing=0). Items from which these data derive differed slightly between the EAPSI Applicant 
Survey and the SDR 2008; thus. Local Government (city, county, school district) and State Government (including state 
colleges/universities) were combined into a single category for both groups; and U.S. Federal Government and U.S. 
MILITARY service, activity duty or Commissioned Corps (e.g., USPFIS, NOAA) were combined for both groups. 

SOURCE: EAPSI Applicant Survey-Items Die, D2 and D3. SDR Survey 2008-Items Al, All. 


Among those working in institutions of higher education, EAPSI applicants were more likely to have 
a position that involved research (81 percent of applicants versus 43 percent from the SDR sample, a 
significant difference) and less likely to be adjunct faculty (1 percent versus 6 percent, a significant 
difference) (Exhibit F.4). 
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Exhibit F.4: EAPSI PhD Applicants and SDR Samples' Positions in Academic Institutions 


President, Provost, or Chancellor 

Dean (any level), department 
head or chair 

Research faculty, scientist, 
associate or fellow 

Teaching faculty 


Adjunct faculty 


Other 



■ EAPSI 

■ SDR 


*p<0.05, **p<0.01, ***p<0.001 

NOTES: The academic positions are not mutually exclusive: Individuals could select more than one response. For example, 
individuals could hold a research faculty position and a teaching faculty position simultaneously. 

These items were answered by EAPSI applicants who had had earned a PhD, were working at an educational institution 
(other than in a postdoctoral position) during the week of October 1, 2010 and who did not report working in a preschool, 
elementary, middle, or secondary school or system (EAPSI N=195 Missing=0) and by SDR respondents who had completed 
a PhD, who were working in an educational institution during the week of October 1, 2008, and who did not report 
working in a preschool, elementary, or secondary school system (SDR N=ll,773 Missing=0). 

SOURCE: EAPSI Applicant Survey-Items Die, D2, D2a and D2c. SDR Survey 2008-Items Al, A12, and A15. 


Similar comparisons of employers were performed for EAPSI applicants who had received master's 
degrees. Applicants were significantly less likely to be self-employed or business owners (1 versus 5 
percent) and less likely to work in the private sector (56 versus 67 percent) (Exhibit F.5). 
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Exhibit F.5: Employers of EAPSI Applicants and NSRCG Sample during Reference Week 


Employed at educational institution 


Self-employed/business owner 


Private sector 


Local/state govnerment 


Federal Government 



■ EAPSI Applicants 

■ NSRCG 


*p<0.05, **p<0.01, ***p<0.001 

Note: Reference week was October 1, 2010 for EAPSI fellows and October 1, 2008 for the NSRCG sample. This item was 
answered by EAPSI awardees who had completed their EAPSI fellowship, whose highest degree earned was a master's 
degree and were employed as of October 1, 2010 (N=323 Missing=0) and by NSRCG respondents who had completed a 
master's degree and were employed as of October 1, 2008. (N=4,326 Missing=0). Items from which these data derive 
differed slightly between the EAPSI Applicant Survey and the NSRCG 2008; thus. Local Government (city, county, school 
district) and State Government (including state colleges/universities) were combined into a single category for both 
groups; and U.S. Federal Government and U.S. Military service, activity duty or Commissioned Corps (e.g., USPFIS, NOAA) 
were combined for both groups. SOURCE: EAPSI Applicant Survey-Items Die, D2 and D3. NSRCG Survey 2008-Items Bl, 
Bll and B12. 
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